OPPO研究院多模态预训练模型在OPPO端云场景的落地实践

DataFunCon#

2023多模态预训练模型在OPPO端云场景的落地实践陈宸-OPPO研究院-高级算法工程师03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11Contents目录端侧图文检索技术研究图文生成&理解模型的应用优化文图生成模型的端侧轻量化03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11端侧图文检索技术研究03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11query1:

和女朋友去迪士尼 query2:

山顶婚纱照一句话搜索的意义：用户体验：真正解决用户想什么就能搜什么的痛点，“智慧搜图，搜你所想”。依托于大模型预训练技术，不再依赖于标签的迭代和扩展大模型轻量端侧化的技术意义：成本节约：将云侧大模型才能体验的效果搬向到端侧，大幅节约计算资源；隐私保护：直接在端侧处理用户的私人照片，无需上传到云端，保护用户隐私；https://b /s端侧图文检索技术研究——解决了什么问题？端侧检索demo实现端侧智慧搜索的关键因素：其一，“人话”解读能力。智慧搜图不仅能单独搜词，也能放一起搜，实现真正的口语化表达搜索，所想即所得，如“去年在动物园拍的老虎”等。因此需要类似多模态大模型

CLIP（OpenAI）的图文理解能力。其二，高效搜索速度。相比动辄翻上十几分钟半个小时的相册，现在无论从桌面下拉智慧搜索、打开相册、或是用语音助手，都只需要一句话就能搜到想要的图片，系统级地提升了找信息的效率。因此如何实现大模型在端侧的轻量化部署有重大的意义。大模型轻量化端侧技术落地的难点：压缩多模态大模型并确保精度。这并非简单用剪枝或量化等方法，直接压缩几倍模型大小就能搞定。毕竟对于端侧而言，算力有限的情况下，能部署的模型大小是往往只能达到大模型的几十分之一。与算法模型升级相对应的，需要在端侧实现一个性能鲁棒的向量检索引擎，保证大模型下端的工程性能03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11端侧图文检索技术研究——算法优化CLIP双塔模型ALBEF单流模型单双流多教师蒸馏架构损失函数检索引擎的计算分位两部分：离线部分：扫描相册所有图片，通过图片编码器将所有图片转成向量；并经过fp16量化存储成Nx200的fp矩阵在线部分：每次输入query，通过文本编码器将query转成向量；并经过fp16量化降低计算内存；最后通过矩阵相乘计算query向量跟所有图片的相似分数，并通过排序输出一个有序列表。Lei,Youbo,etal."MCAD:Multi-teacherCross-modalAlignmentDistillationforefficient

image-textretrieval."arXivpreprintarXiv:2310.19654

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11端侧图文检索技术研究——学术集效果各种蒸馏方法的效果对比Modelnameimagemodeltextmodelfusion

modelimage

encodingtimeretrieval

timeparameternumbertestsetplatformCLIPVIT-L/1412-layertransformerdot

product11.0ms32.5ms427.62Mfilckr5KV100

GPUALBEFVIT-B/166-layertransformer6-layertransformer7.6ms265ms(k=16)1945ms

(k=128)3865ms

(k=256)419.12Mfilckr5KV100

GPU自研小模型mobileVitV2-1.54-layerTinyBertdoc

product3.8

ms14.1

ms25.9

Mfilckr5KV100

GPU自研小模型mobileVitV2-1.54-layerTinyBertdoc

product17.3

ms14.6

ms25.9

Mfilckr5KMTK

DX3大小模型的性能对比03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11端侧图文检索技术研究——真实场景效果数据量：11个用户真实相册共2万+图片，手写5400+query数据分布：测试集R@1R@5R@10MRmAP010.47280.6710.74950.63110.6080020.49560.7580.82510.69290.5306030.40190.56650.61080.52640.4889040.45320.68470.73890.62560.6048050.58430.7530.79520.71080.6428060.53230.68550.750.65590.5890070.350.52940.60880.49610.4771080.64170.80830.84170.76390.5943090.59650.68420.71930.66670.5622100.51210.70590.76470.66090.5441110.56540.74180.7810.69610.6336平均0.48480.67680.73600.63250.584003872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11端侧图文检索技术研究——细粒度优化Doveh,Sivan,etal."Teachingstructuredvision&languageconceptstovision&languagemodels."ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecognition.2023.细粒度属性词替换+hard

negative

sampling+

LwF抗遗忘算法03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11文图生成&理解态模型的应用优化03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——中文文生图大模型继续预训练如何做高质量低成本的继续预训练如何对齐中文的语言文化如何提升生成图像的细节质量Parameterefficient

adapterOrthogonal

FinetuningQiu,Zeju,etal."Controllingtext-to-imagediffusionbyorthogonalfinetuning."Thirty-seventhConferenceonNeuralInformationProcessingSystems.

2023.03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11中文语境迁移效果图文生成&理解模型的应用优化——中文文生图大模型继续预训练收敛速度03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11Finetune模型鸳鸯双栖蝶双飞，满园春色惹人醉LoRAControlnetSSD1.3B小模型一只超级可爱的兔子穿着僧侣服装，肖像照，皮克斯动画SDXLinpainting青花瓷版的恐龙在长椅上江南，夏日湖畔的一个村庄图文生成&理解模型的应用优化——中文文生图大模型继续预训练一个漂亮的亚洲女孩，电影灯光 西湖，塔和瀑布，日出3D电影，4k，高度细致，男人坐在马桶上读报带着墨镜的猫咪手里拿着剑，在恶魔城堡里，仙剑奇侠风格LatentCM03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——通用优化应用壁纸生成春节档热度top1春节档热度top3文生图模型+超分辨率生成2k高清壁纸03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——通用优化应用锁屏杂志生成文生图模型+微调LLAVA+LLM

生成图文并茂的杂志Liu,Haotian,etal."Visualinstructiontuning."arXivpreprintarXiv:2304.08485

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——通用优化应用Zhang,Pan,etal."Internlm-xcomposer:Avision-languagelargemodelforadvanced

text-imagecomprehensionandcomposition."arXivpreprintarXiv:2309.15112

(2023).Internlm-xcomposer训练框架03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域AI模型画人的几个问题：1.

人脸人手等身体部位的崩坏。2.

过于精致标准，渲染过度光滑，在质感上失真。3. 细粒度属性和文本描述的不对齐。03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域构建细粒度的人像属性数据03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域U-Net中模块与图像中特征的对应关系，可用于指导LoRA微调的参数选择厚嘴唇薄嘴唇03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域小鼻子大鼻子03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域细眉毛粗眉毛03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-人像垂域垂域微调经验：大量数据粗调，增加模型对新概念的泛化能力少量高质量数据精调，提升模型的图片生成质量人脸修复逻辑：穿着华丽盔甲的玄幻战士与巨龙激战，雷霆与火焰交织在一起。（随机6张，无cherry-pick）03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化-古风人像效果古道边一骑红尘客正巍然马背，身披白色斗篷，踏寂静落阿叶（随机6张，无cherry-pick）树丛中，翩翩少女，红衣绿裙，手提花伞，踏泥寻径，仿佛踏入了一幅画卷（随机6张，无cherry-pick）03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——垂域优化应用广告营销工具（内测版）03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——文字渲染-问题定义如何在文生图模型中渲染出正确的文字？Ma,Jian,etal."GlyphDraw:LearningtoDrawChineseCharactersinImageSynthesisModelsCoherently."arXivpreprintarXiv:2303.17870

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——文字渲染-算法GlyphDraw训练框架GlyphDraw推理框架数据集图文对数量文字数量中文数据集792k3.3M

字英文数据集1.9M2.3M

wordsMa,Jian,etal."GlyphDraw:LearningtoDrawChineseCharactersinImageSynthesisModelsCoherently."arXivpreprintarXiv:2303.17870

(2023).GlyphDraw数据集构建03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——文字渲染-客观效果Ma,Jian,etal."GlyphDraw:LearningtoDrawChineseCharactersinImageSynthesisModelsCoherently."arXivpreprintarXiv:2303.17870

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——文字渲染-主观效果Ma,Jian,etal."GlyphDraw:LearningtoDrawChineseCharactersinImageSynthesisModelsCoherently."arXivpreprintarXiv:2303.17870

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-问题定义Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).如何使用一张参考图像快速生成新图片并平衡保真度和泛化性？03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-数据集SDD数据集统计数据SDD数据集词云Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-算法Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-效果单实体生成与各种方法的对比Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).双实体生成与各种方法的对比03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-效果Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-效果Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-效果Ma,Jian,etal."Subject-diffusion:Opendomainpersonalizedtext-to-imagegenerationwithouttest-timefine-tuning."arXivpreprintarXiv:2307.11410

(2023).03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成应用外观多角度生成品牌调性干预产品外观描述生成根据参照图生成效果图描述生成品牌调性/风格干预广告营销工具产品外观设计（从0-1设计） 产品效果图生成（工作室拍摄的效果图）产品营销素材生成（海报/banner)营销文案&图片生成素材布局生成布局描述生成参照物干预设计草图生图Ayellow

hatAgirlwearingthehatandfacing

forest选择生成【海报】根据参考素材生成根据品牌VI，历史产品调性生成产品设计03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-应用商品设计个性化图片生成海报设计03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-应用Subject-diffusion的个性化生成03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11图文生成&理解模型的应用优化——个性化生成-应用Stable-diffusion的outpainting03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11文图生成模型的端侧轻量化03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11文图生成模型的端侧轻量化——技术路线-模型结构优化Unet结构示意图删除某个模块之后的效果和参数量波动分析模型采样时间（DPMsolver++25步）运行内存UNet参数量SD

1.51.34s4105M859.52MSD

base-2m0.9s3458M579.38MSD

small-2m0.83s3287M482.35MSD

tiny-2m0.76s2979M323.38MSD

small0.88s3477M579.38MSD

tiny0.75s3043M323.38M不同剪枝模型在V100上测试结果03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11文图生成模型的端侧轻量化——技术路线-模型结构优化采用SDXL蒸馏SD

small模型03872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11-29803872412023-11文图生成模型的端侧轻量化——技术路线-采样加速Salimans,Tim,andJonathanHo."Progressivedistil