讲稿大数据-学习phpapp

1、Redis And PythonPyCon India, 2011Sunil AroraRaising Hands.How many of you have used Redis before ?How many of you have got a laptop ?About MeI tweet atSunil Arora / _sunil_I work atShopSociallyI blog at Todays talkWhat is RedisHow it works and what you can do with itReal life use-casesReal life use-

2、casesSome hand-on with RedisRedis - Brief HistoryInitially written to improve performance of Web Analytics product LLOOGG out of his startupSalvatore Sanfilippoantirez Redis Brief HistoryReleased in March 2009 (Open Source, BSD licensed)VMWare hired Salvatore in March, 2010Then Pieter Noordhuis (key

3、 contributor) was hiredWhat is Redis ?Its between lot of stuff, so difficult to categorize it preciselyWhat is Redis ?I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well-S

4、alvatore SanfilippoPicture by herzogbr Redis is.Remote Data Structure ServerRedis is.Supports rich data types of computer science- Strings, Lists, Sets, Sorted Sets, Hashes.Rich sets of primitives (commands) to manipulate these types Predictive complexity measurementsA few fundamentalsWritten in C (

5、no external dependency)Uses memory as main storageUses disk for persistenceSingle ThreadedEvery command performs atomic execution PerformanceScreamingly fast performance50K read/write operations per seconds100K+ read/write ops per second on a regular EC2 instance Installation$ git clone $ cd redis$

6、make$ ./src/redis-server.$./src/redis-cliRedis PINGPONGPython LibrariesRedis-py - Python client libraryTxredisapi - An asynchronous Python client for the Redis database, based on Twisted.Redisco - ORM for redis along the lines Ohm for Rubyredis-pyThe most popular python client libraryAndy McCurdy (

7、)Github: $easy_install redis OR$pip install redisOptional$easy_install hiredis$pip install hiredisLets get started.$ cd redis$ ./src/redis-server. from redis import Redis redis_client = Redis() redis_client.keys() help(redis_client)Redis KeysNot binary safe.Should not contain space or newline charac

8、terA few rules about keys:Too long keys are not a good ideaToo short keys is also not a good idea“object-type:id:field” can be a nice idea, i.e. “user:1001:name”Operations on KeysKEYSEXISTSDELEXPIREOBJECTPERSISTRANDOMKEYRENAMETYPETTLEXPIREATMOVELets play with keysredis_client.keys()redis_client.exis

9、ts(key)redis_client.delete(key)redis_client.type(key).Data StructuresStringsListsSetsSorted SetsHashesStringsSETGETMSETMGETSETEXGETSETSETNXINCRINCRBYDECRDECRBYStrings with redis client redis_client.set(key, value) redis_client.get(key) redis_client.delete(key)Fetch multiple keys at oncemget/msetredi

10、s_client.mset(key1: val1, key2: val2)redis_client.mget(key1, key2,.)ExpirationSet a value with expireredis_client.setex(key, value, 2) #key to expire in 2 secsredis_client.expire(key, 2)redis_client.get(key)NoneExpire SemanticsKey with expiry known as volatile keysWhenever a volatile key is modified

11、, its expiry is reset - To avoid inconsistency in following casesReplicationAppend Only LogUsesTo store transient states in your web applicationUsesWho is online?UsesRedis as LRU cache ( )Atomic Incrementshelp(redis_client.incr)help(redis_client.decr) redis_client.incr(counter, 1)1 redis_client.incr

12、(counter)2 redis_client.incr(counter)3UsesHigh Speed counters (views/clicks/votes/likes.)UsesAPI Rate LimitingUsesGenerating unique IDsListsOrdered list of binarysafe stringsDoubly linked listMemory footprint optimized for smaller listO(1) insertion/deletion at both endsLists - operationsLPUSHRPUSHL

13、SETLRANGELPOPBLPOPBRPOPBRPOPLPUSHLINSERTRPOPRPOPLPUSHLPUSHXRPUSHXUsesWeb apps are full of lists :)UsesCapped ListUsesReal time message QueueBackground Worker queues (Resque, Celery) UsesSocial Activity Streams or notificationsSetsAn unordered collection of distinct byte stringsNothing different from

14、 the data type in pythonSets - OperationsSADDSCARDSREMSISMEMBERSMEMBERSSPOPSRANDMEMBEROperations between SetsSMOVESUNIONSDIFFSINTERSets - OperationsSDIFFSTORESINTERSTORESUNIONSTORESets - UsesPicking random items from a set using SRANDMEMBEREx.Picking a random article from daily newsPick a random Ad

15、for servingPick a random option for A/BNote: Time complexity is O(1). Compare it with SQLs “order by Rand()”Sets - UsesTo model Relations in social graphEx.redis_client.sadd(friends:john, jenny, maria)redis_client.sadd(friends:ben, maria, kate)redis_client.sinter(friends:john, friends:ben)Relations

16、(friend/followers)Common followlist for A and Bsinter(users:A:follows, users:B:follows)Unique to B compared to Csdiff(users:B:follows, users:C:follows) Mutual relationship (friend as well as follower)sinter(users:B:followers, users:B:friends)Who does not follow me back ?sdiff(users:B:friends, users:

17、B:followers)Who am I not following back?sdiff(users:B:followers, user:B:friends)Which of my friends are online right now? SINTER online_people my_friendsWhich of my friends are online right now?RENAME online:fresh online:stale #every minuteSUNIONSTORE online:fresh online:stale#friend jack connectsSA

18、DD online:fresh jackSUNIONSTORE online online:fresh online:stale#who is online ?SINTER online friendsSorted SetsOrdered sets on the basis of scoreSorted SetsZADDZCARDZCOUNTZINCRBYZINTERSTOREZRANGEZRANGEBYSCOREZRANKZREMZREMRANGEBYRANKZREMRANGEBYSCOREZREVRANGEZREVRANGEBYSCOREZSCOREZUNIONSTORESorted Se

19、ts Use CaseTo build index on your datasetUses - Realtime Leaderboardszincrby(leaderboard, john, 2)zincrby(leaderboard, jack, 5)zincrby(leaderboard, kate, 1)zincrby(leaderboard, kate, 10).zrange(leaderboard, 0, -1, withscores = True)Uses - Realtime LeaderboardsMost download resourcesMost popular arti

20、cles on the websiteWeekly popular listMost downloaded stuff between date1 and date2. plete with Sored Sets HashesEquivalent to Python dictionary or Ruby hash or Java hashmapOperationshmset users:1 username: jim, score: 23hgetall users:1HincrbyUseful for storing structured datahmset user:1:preference

21、s flash_shown: yes, bgcolor: #fffExpire user:1:preferences Hmgetall user:1:preferences #returns entire dict Redis TransactionsUsing Multi/Watchp = redis_client.pipeline()p.lpush(a, 1)p.ltrim(a, 0, 100)p.executeLimitationSince commands are queued, Cant read valuesNo conditional executionIf not high write contention scenario, WATCH can be usedUse Redis Scripting to write your own primitiveDesigning with RedisData layout design on basis of QueryNo Query OptimizerManually build indexesDurabilitySnapshotting modeBinary dump every x secs or y opsAppen