无服务器架构设计模式和最佳实践

1、无服务器架构设计模式和最佳实践技术创新 变革未来AWS所提供的服务类型AWSLambdaAmazonKinesisAmazon S3Amazon APIGatewayAmazon SQSAmazon DynamoDBAWS IoTAmazon EMRAmazon ElastiCacheAmazon RDSAmazon RedshiftAmazon Elasticsearch ServiceServerlessAmazon EC2Microsoft SQL ServerAmazon CognitoAmazonCloudWatch开源服务自动化、自有服务开放化。现在？无服务器化！“On EC2”M

2、anagedAWS Serverless 客户S3 Bucket(s)Amazon AthenaKeyAWSManagement CloudTrail ServiceAmazon CognitoAWS IAMAmazon Kinesis StreamsAmazon Kinesis FirehoseAmazon QuickSightAWS GlueAmazon ESAmazonDynamoDBAmazon MacieAmazon API GatewayAWS IAMAmazon Redshift SpectrumAWS Direct Connect摄入Catalog & 搜索安全 & 审计API

3、/UI分析 & 处理AWS GlueAWSLambdaAWS Serverless Data LakeAWS Lambda客户案例 - YublAWS Lambda 客户案例 - YublA Lambda function that processes Kinesis events should also:retry failed events X times depending on processing timesend failed events to a DLQ after exhausting X retries/2017/04/aws-lambda-3-pro-tips-for-w

4、orking-with-kinesis-streams/设计模式: Lambda优化Amazon API GatewayAmazon AlexaAWS IoTAmazon KinesisAmazon SNSAmazon SESAWS Step Functions213Invocations Functions InteractionsAmazon S3Amazon DynamoDBCustom endpointsAmazonAmazonElasticsearch CloudWatchEC2 instanceAWS Lambda 优化Your functionLanguageruntimeExe

5、cution EnvironmentCompute substrateLambda函数无服务器的设计模式建立在Functions之上Functions是最小的部署和扩展单元按照请求来自动扩展，用户无需为这些关注这些扩展的开销无需为idle的请求付费跳过了最无聊的部分，也是越过了最难的部分AWS Lambda并发问题 - 流量激增问题增加您的并发执行函数/zh_cn/lambda/latest/dg/concurrent-executions.htmlAWS Lambda 将根据增加的流量动态扩展容量，具体取决于您的AWS账户的并发执行限制。为了处理突增流量，Lambda 将立即根据预定量增加您

6、的并发执行函数。3000直到达到账户的安全限制， 或并发执行的函数数量足 以成功处理增加的负载Lambda函数时间轴（Lambda不在VPC里面）Bootstrapthe runtimeStart yourcodeFull cold startPartial cold startWarm startDownloadyour codeStart newcontainerAWS optimizationYour optimizationAWS X-Ray跟踪Lambda冷启动时间Before the function handler code executes, there is an also

7、additional 200ms of “Initialization”thats the time it takes the function to load its dependencies and run code outside of the handler function.AWS X-Ray跟踪Lambda冷启动时间In the second “warm start” case, the function handler code starts almost immediately after the service receives the request to invoke t

8、he functionand no initialization occurs. Minimizing time spent in the Lambda service and avoiding function initialization on requests can have significant performance implications.Lambda函数代码Avoid “fat”/monolithic functionsControl the dependencies in your functions deployment packageSeparate Lambda h

9、andler (entry point) from core logicOptimize for your languageNode Browserfy, MinifyLambda的注意事项和最佳实践你的Lambda Function能在冰天 雪地中幸存下来么?(Code Start)要注意初始化AWS客户端连接或 者数据库连接的时候的变量scope， 尽可能的复用连接利用CloudWatch Events来做预热ENIs for VPC将会在Code Start时 候被加载import sys import loggingimport rds_config import pymysqlrds

10、_host = rds-instance db_name = rds_config.db_name try:conn = pymysql.connect( except:logger.error(ERROR: def handler(event, context):with conn.cursor() as:cur仅在调用时被执 行冷启动时执行流失处理针对敏感的延时问题 - Cold StartLambda是否使用VPC？Should my Lambda function be in a VPC?Does my function need to access any specific reso

11、urces in a VPC?Does it also need to access resources or services in the public internet?Dont put the function in a VPCPut the function in a private subnetPut the function in a subnet with a NATd route to the internetYesYesNoNoLambda配合VPC的时间轴（Lambda在VPC内）Downloadyour codeStart newcontainerStart yourc

12、odeCreateAttachVPC ENI VPC ENIFull cold startWarm startBootstrapruntimePartial cold startAWS optimizationYour optimizationALWAYS configure a minimum of 2 Availability ZonesGive your Lambda functions their own subnetsGive your Lambda subnets a large IP rangeto handle potential scaleIf your functions

13、need to talk to a resource onthe internet, you need a NAT!Lambda的注意事项和最佳实践 - Lambda in VPC了解重试机制同步模式不会重试异步模式重试两次Stream-Base模式会一直重试使用Dead Letter Queues（私信队列）使用SQS 或者 SNS 来重试记住: 重试次数也算一次调用Lambda“重试”机制Lambda的注意事项和最佳实践文件系统如何考量?别忘记我们还有/tmp (512 MB 空间)exports.ffmpeg = function(event,context)new ffmpeg(./t

14、humb.MP4, function (err, video)if (!err) video.fnExtractFrameToJPG(/tmp) function (error, files) if (!error) console.log(files); context.done();.设计模式: 批处理批处理的特征大批的数据量定期的或者被计划的任务Extract Transform Load (ETL)的工作模式大部分情况下不会长期运行大部分问题都能用MapReduce的模式来解决无服务器架构 批处理AWS Lambda: SplitterAmazon S3 ObjectAmazon S3

15、:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 Results无服务器架构 批处理AWS Lambda: SplitterAmazon S3 ObjectAmazon DynamoDB:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 ResultsAWS Lambda处理map-reduce类型工作Lambda Counting Function 0S3 BucketUsersLambda Counting Function 0Lambd

16、a Counting Function nSource DataSource DataLambda CountingFunction nSource DataSource DataLambda CascadeFunction 0Lambda CascadeFunction nNode.js Web AppRequest ResponseRequestResponseRequest ResponseRequest ResponseRequest ResponseRequest ResponseQuery responds with URL for the event stream. App in

17、vokes Lambda and stream resultServer-Sent Event streaming responseWeb UICascadeProcessQueryAWS中国（北京）区域由光环新网运营注意事项和最佳实践串联mapper functionsLambda批处理模式 vs. SQL (Hadoop / Spark生态)处理数据和Lambda function的并发和限制需要取得平衡使用DynamoDB/ElastiCache/S3来管理mapper functions的状态Lambda MapReduce Reference ArchitectureLambda已经

18、集成了SQS事件触发Lambda已经集成了SQS事件触发设计模式: 流式处理流式处理的注意事项高吞吐近乎实时的处理能力爆发性的间歇性流量消息持久化消息处理的保序无服务器架构流式处理SensorsAmazon Kinesis:StreamLambda:Stream ProcessorS3:Final Aggregated OutputLambda: Periodic Dump to S3CloudWatch Events: Trigger every 5 minutesS3:Intermediate Aggregated DataLambda: Scheduled DispatcherKPL:P

19、roducerLambda流式处理并发问题事件源Amazon Kinesis目的地 1Lambda目的地 2ShardsScale Amazon Kinesis by splitting or merging shardsPolls a batch Waits for responseLambda 自动扩展一个 Kinesis Shard:If no record, wait some time (1s)Sub-batch in-memory and format records into Lambda payloadInvoke Lambda with synchronous invokeL

20、ambda blocks on ordered processing for each individual shardIncreasing # of shards with even distribution allows increased concurrencyBatch size may impact duration if the Lambda function takes longer to process more recordsSourceAmazon KinesisDestination 1ShardsScale Amazon Kinesis by splitting or

21、merging shardsDestination 2Lambda 自动扩展LambdaPolls a batWchaits for responseLambda流式处理并发问题数据源ShardsTrim horizonCheckpointLatestCheckpointLambda流式处理并发问题Event received by Lambda function is a collection of records from the stream: Records: kinesis: partitionKey: partitionKey-3,kinesisSchemaVersion: 1.0

22、,data: SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=,sequenceNumber: 49545115243490985018280067714973144582180062593244200961 ,eventSource: aws:kinesis,eventID: shardId-000000000000:49545115243490985018280067714973144582180062593244200961,invokeIdentityArn: arn:aws:iam:account-id:role/testLEBRole,eventVersio

23、n: 1.0, eventName: aws:kinesis:record,eventSourceARN: arn:aws:kinesis:us-west-2:35667example:stream/examplestream,awsRegion: us-west-2 Lambda流式处理并发问题Lambda流式处理并发问题Lambda流式处理Polls and blocks on synchronous invocation per shardIf put/ingestion rate is greater than the theoretical throughput, your proc

24、essing is at risk of falling behind Maximum theoretical throughput # shards * 2 MB / Lambda function duration (s)Effective theoretical throughput # shards * batch size (MB) / Lambda function duration (s)Retries 流失处理的重试机制Will retry on execution failures until the record is expiredThrottles and errors

25、 impact duration and directly impact throughputBest practiceRetry with exponential backoffUse case for service integrationRealtime clickstream logs to Kinesis streamReal-time Clickstream Anomaly Detection Reference:/cn/blogs/big-data/real-time-clickstream-anomaly-detection-with-amazon-kinesis-analyt

26、ics/扇出模式（fade-out）Kinesis shards的数量等同于Lambda的调用并发数量高吞吐和低延时 vs 消息保序SensorsAmazon Kinesis:StreamKPL:ProducerLambda: ProcessorsIncrease throughput, reduce processing latency如果Kinesis在单个shard处理的时候超过了这个限制 级的处理能力，就需要考虑使用Fade out的模式了。分发者 Lambda: Dispatcher缺点？ 保序性！最佳实践调教Kinesis每次调用 Lambda触发器batch size的 值，默认

27、是100，更大的值 能够使得Lambda有更少的 调用次数调教Lambda的内存 更快的处理速度尽量使用KPL的Batch Message功能来批量发送数 据，这样可以更充分的利用Kinesis的高吞吐特性Lambda architectureData SourcesServing LayerSpeed LayerBatch Layer模式4: 自动化处理自动化处理所具备的特征处理报警和各类事件周期性的计划任务审计和通知对AWS服务功能性上的扩展高可用 + 自动扩展自动化处理: 动态为EC2分配DNSAWS Lambda: Update Route53Amazon CloudWatch Eve

28、nts: Rule TriggeredAmazon EC2 Instance State ChangesAmazon DynamoDB:EC2 Instance PropertiesAmazon Route53: Private Hosted ZoneTag:CNAME = A 34/awslabs/route53-dynamic-dns-with-lambda自动化处理: 从S3加载生成图片缩略图AWS Lambda:Resize ImagesUsers upload photosS3:Source BucketS3:Destination BucketTriggered on PUTs图片

29、打水印（Lambda + S3）/bjsdemo/Lambda-watermark-demo/Lambda_watermark_demo.zip图片压缩/裁剪（Lambda + S3）/bjsdemo/Lambda-CreateThumbnail-demo/CreateThumbnail.zip根据输入参数线上裁剪图片（API Gateway + Lambda + S3）/bjsdemo/Serverless-Image-Resizing-Demo/serverless-image- resizing-demo-Danrong.zip自动化处理 - 配合Lambda还有很多常用场景EC2实例定时开机、关机：/awslabs/ec2-scheduler/blob/master/code/ec2-scheduler.pyEBS定时打快照，并记录打快照的是否成