为社交网站构建新型数据产品(John Wang)

v

Qcon–Hangzhou,October2011JohnWang–SearchArchitect

BuildingInnova@veDataProducts

1

Agenda

•Introduc@on

–LinkedIn

–Myself

–Dataproducts

•Understandingyourdata

•Technologybehindit

•Take-aways

AboutLNKD

•Founded2003–HQ:MountainView,CA

•LargestProfessionalSocialNetwork

•120MillionUsers

–~50%US

–170+countries*(asoflastyear)

•IPO,May2011(nolongerastartup,buts@llastartupatheart)

•Oneofourbiggestassets:OurData

SpeakerInfo

•JohnWang(王鉴)

•SearchArchitect@LinkedIn

•JoinedLinkedInin2007

•FromShanghai,China

•Twi_er:@javasoze

•Email:john.wang@

Dataproducts

•Matchingdatatouserintent

•Exposevalueindata

•Examples

PeopleSearch

/search

/signal

Signal

LinkedInToday

/today

RecommendationEngine

/jsearch/rec

Understandingyourdata

•Volume

•Productvaluesbehindit

Volume:InformationExplosion

13

“Moredatawillbecreatedinthenextfouryearsthaninthehistoryoftheplanet.”

MarkHurd,CEOofHP,June,2009

14

WebPagesIndexed(inMM)

3,000X

200

20

60,000

8,000

1995199920052010

ProductValues

•Whatis“relevance”

•Importanceofcontext

•Ahamoment

•Socialdata

17

Visa717Justmadeblueberrycheesecakeicecreamandnowit’stimeforpoker!

1hoursagoviatxt

mygurlgotAgfThisoldwomanjusthitonme

1hoursagoviatxt

WSJApplesurpassedMicrosofttodayasthemostvaluableTechnologycompanywithamarketcapof#223billion

/cTVY6x

#in

2hoursagoviaLinkedIn

De00Finishedwatching2012.WassooooootankJ

1hoursagoviatxt

18

WSJApplesurpassedMicrosofttodayasthemostvaluableTechnologycompanywithamarketcapof#223billion

/cTVY6x

#in

2hoursagoviaLinkedIn

19

20

Technologybehindit

•Choicesofarchitecture

•Querytypes

•Specializedstorage

ChoicesofArchitecture

•3-@ered(MVC)architecture

•Offline->publish->online(Google)

–Separa@ngreadandwriteopera@ons

•Streamedbased

DataWarehouse

•HighVolume

•Highlatencyacceptable

•Lowqueryvolume

•Learningyourdata

•Trend:movingtospecializedstores

Querytypes

•get/getAll

•Filter

•Sort

•Aggregate

•Grouping

•Relevance?

SpecializedDatastore

•Get/GetAll/Put

–Voldemort(AmazonDynamo)

–DistributedK-Vstores

•Sensei

–Semi-structuredqueryengine

–Open-source:h_p://

–Firstreleasecomingsoon…

Sensei

•Exis@ngtechnology

–Cassandra

–Hbase(GoogleBigTable)

–Solr/Elas@cSearch

–Hive/Pig

•SupportedQueries

•Transac@onalGuarantees

•Overview

Sensei-Queries

•HardFilter(whereclause)

•SoqFilter(full-textquery)

•DynamicSor@ng

–Field-based

–Relevance-based

•Paging(limit)

•Grouping

•Face@ng

•Adjustablerelevancemodel

SenseiQueryExample

Selectfrommemberswhereindustry=IT

ANDsocial-network(‘jwang’,FIRST|SECOND)AndqueryIS‘@tle:engineer^5ORsummary:java’ORDERBYrelevance

LIMIT10,10

GROUPBYloca@on

FACETBYcompany,industry,social-networkRELEVANCE_FUNCTION(‘relevance-model-1’)

Transac@on(ACID)Guarantees

•Streambasedupdates(orderedandversioned),guaranteesAtomicityandIsola@on

•Replicated–durability

•Eventualconsistency-Eachreplicaconsumingfromthesameversionedstream

Overview

Overviewinternals

•Log-basedpersistence

–Immutablesegments,periodicmerge

–Op@mizeforbalancedsegmentsize

•Invertedindexandcompressedforwardindexinmemory

•Pluggable:relevancemodel,filter,aggregator,sorter

•Schemadriven

Overview(cont’d)

•Sub-secondlatency(10’sofms)

•Fastupdates(1000sups)

•Datajoinswithexternalstoragesystem,e.g.oursocialgraphengine

•Minimizeopera@onscost

•Produc@on-ready(>1yearofproduc@onuse)

•PoweringLinkedInhomepage

SomeTa