ST-Net: Tsarin Mai Gudanar da Kai don Haɗa Tufafi Masu Jituwa Ba tare da Kulawa ba

2.2. Tsarin ST-Net

ST-Net an gina shi akan tsarin Cibiyar Sadarwa ta Halitta (GAN). Babban ƙirƙirarsa shine mai shigar da hanya biyu wanda ke bayyana raba hoton da aka shigar zuwa lambar salo $s$ da lambar nau'i $t$.

• Mai Shigar da Salo: Yana fitar da manyan fasalulluka na ma'ana, na duniya (misali, "bohemian", "minimalist").
• Mai Shigar da Nau'i: Yana ɗaukar ƙananan fasalulluka, na gida (misali, plaid, polka dots).

2.3. Dabarun Koyo Mai Gudanar da Kai

Don horarwa ba tare da biyu ba, ST-Net tana amfani da dabarar da aka yi wahayi zuwa ga daidaitaccen zagayowar amma tana daidaita shi don dacewa da matakin halaye. Babban ra'ayi shine musanya halaye da sake gina. Don abubuwa biyu marasa biyu $(x_i, y_j)$, an fitar da lambobin salo da nau'i. An ƙirƙiri "na zahiri" biyu masu dacewa ta hanyar, misali, haɗa salo na $x_i$ tare da nau'i daga yankin manufa. An horar da cibiyar sadarwa don sake gina abubuwan asali daga waɗannan wakilcin da aka musanya, yana tilasta masa koyon ma'ana mai ma'ana da wakilcin dacewa.

3. Cikakkun Bayanai na Fasaha

3.1. Tsarin Lissafi

Bari $E_s$ da $E_t$ su zama masu shigar da salo da nau'i, kuma $G$ ya zama mai ƙirƙira. Don hoton da aka shigar $x$, muna da: $$s_x = E_s(x), \quad t_x = E_t(x)$$ Tsarin ƙirƙira don abu mai dacewa $\hat{y}$ shine: $$\hat{y} = G(s_x, t')$$ inda $t'$ shine lambar nau'i, wanda za'a iya samun samfurinsa, samo shi daga wani abu, ko kuma a koya shi azaman canji na $t_x$ don dacewa da yankin manufa.

3.2. Ayyukan Asara

Jimlar asara $\mathcal{L}_{total}$ haɗe ne na manufofi da yawa:

• Asarar Adawa ($\mathcal{L}_{adv}$): Asarar GAN na yau da kullun yana tabbatar da gaskiyar fitarwa. $$\min_G \max_D \mathbb{E}_{y \sim p_{data}(y)}[\log D(y)] + \mathbb{E}_{x \sim p_{data}(x)}[\log(1 - D(G(x)))]$$
• Asarar Sake Gina Kai ($\mathcal{L}_{rec}$): Yana tabbatar da cewa masu shigar da bayanai sun ɗauki isassun bayanai. $$\mathcal{L}_{rec} = \|x - G(E_s(x), E_t(x))\|_1$$
• Asarar Daidaiton Halaye ($\mathcal{L}_{attr}$): Babban ƙirƙira. Bayan musanya halaye (misali, amfani da salo daga $x$ da nau'i daga bazuwar $y$), cibiyar sadarwa yakamata ta iya sake gina asalin $y$, yana tilasta cewa abin da aka ƙirƙira ya riƙe halayen da aka musanya. $$\mathcal{L}_{attr} = \|y - G(E_s(x), E_t(y))\|_1$$
• Asarar Rarrabuwar KL ($\mathcal{L}_{KL}$): Yana ƙarfafa sararin samaniya da aka raba (salo/nau'i) don bin rarrabuwar farko (misali, Gaussian), yana inganta gabaɗaya.

4. Gwaje-gwaje & Sakamako

4.1. Bayanan Gwaji

Marubutan sun gina babban bayanan CCS mara kulawa daga tushen yanar gizo, wanda ya ƙunshi ɗaruruwan dubban hotunan tufafi na sama da ƙasa marasa biyu. Wannan yana magance babban matsalar bayanai a fagen.

4.2. Ma'aunin Kimantawa

An kimanta aikin ta amfani da:

• Makin Shiga (IS) & Tazarar Shiga Fréchet (FID): Ma'auni na yau da kullun don ingancin ƙirƙirar hoto da bambancin.
• Makin Dacewar Tufafi (FCS): Ma'auni da aka koya ko kimantawar ɗan adam wanda ke kimanta yadda abin da aka ƙirƙira ya dace da abin da aka shigar ta fuskar salo.
• Nazarin Mai Amfani (Gwajin A/B): Alkalan ɗan adam sun fi son sakamakon ST-Net fiye da hanyoyin tushe dangane da dacewa da gaskiya.

4.3. Sakamako na Ƙididdiga & Halaye

Ƙididdiga: ST-Net ya sami mafi kyawun maki FID da IS idan aka kwatanta da hanyoyin I2I marasa kulawa na zamani kamar CycleGAN da MUNIT, yana nuna ingancin hoto mafi kyau. Haka kuma ya fi su gaba sosai akan Makin Dacewar Tufafi.
Halaye: Sakamakon gani yana nuna ST-Net ya yi nasarar ƙirƙirar ƙasan da ke da salo masu daidaituwa (misali, na kasuwanci na yau da kullun) da nau'i (misali, layi masu dacewa ko palettes launi) tare da saman da aka shigar. Akasin haka, hanyoyin tushe sau da yawa suna samar da abubuwa waɗanda suke da gaskiya amma sun yi karo da salo ko kuma sun kasa canja wurin mahimman siffofi.

5. Tsarin Bincike & Nazarin Hali

Babban Fahimta: Babban nasarar takarda ba wani bambancin GAN kawai ba ne; yana da tunani na asali game da matsalar "dacewa". Maimakon ɗaukar shi azaman fassarar matakin pixel (wanda ya gaza saboda rashin daidaitawar sarari), sun sake tsara shi azaman ƙirƙirar yanayi na matakin halaye. Wannan hanya ce mai hikima, mafi kama da ɗan adam ga AI na tufafi.

Kwararar Hankali: Hankali yana da kyau: 1) Amincewa bayanai biyu matsala ce. 2) Gano cewa salo/nau'i, ba siffa ba, ke motsa dacewa. 3) Ƙirƙirar cibiyar sadarwa wanda ke bayyana raba waɗannan halayen. 4) Amfani da gudanar da kai (musanya halaye) don koyon aikin dacewa daga bayanan da ba a haɗa su ba. Wannan kwararar ta kai hari kai tsaye ga matsalolin babban matsala.

Ƙarfi & Kurakurai:
Ƙarfi: Dabarar raba bayyana tana da fassara da tasiri. Gina babban bayanan da aka keɓe babbar gudummawa ce mai amfani. Hanyar tana da ƙarfi fiye da hanyoyin da suka dogara da biyu.
Kurakurai: Takarda ta nuna amma ba ta warware matsalar "rashin fahimtar salo" ba—yadda ake ayyana da ƙididdige "salo" fiye da nau'i? Kimantawa, ko da yake an inganta shi, har yanzu yana dogara da wani ɓangare akan makin ɗan adam na zahiri. Hanyar na iya fuskantar wahala tare da canja wurin salo mai zurfi ko na gaba inda ƙa'idodin dacewa ba su da ƙayyadaddun bayyana.

Fahimta Mai Aiki: Ga masu aiki: Wannan tsarin shiri ne don motsawa fiye da AI na tufafi mai kulawa. Dabarar gudanar da kai ta musanya halaye tana aiki ga sauran yankuna kamar ƙirar kayan daki ko kayan ado na cikin gida. Ga masu bincike: Gaba gaba shine haɗa siginonin nau'i-nau'i (bayanin rubutu na salo) da kuma matsawa zuwa cikakken ƙirƙirar kayan sawa (kayan kwalliya, takalmi) tare da keɓancewar mai amfani a cikin madauki. Aikin masu bincike a MIT's Media Lab akan hankalin kyan gani yana ba da jagora mai dacewa don ayyana salo ta hanyar lissafi.

6. Aikace-aikace na Gaba & Jagorori

• Mataimakan Tufafi Na Sirri: An haɗa su cikin dandamalin kasuwanci na e-commerce don shawarwarin "cika kallon" na ainihin lokaci, yana ƙara girman kwandon kwandon.
• Tufafi Mai Dorewa & Ƙirar Lambobi: Masu zane-zane na iya ƙirƙirar tarin abubuwa masu dacewa cikin sauri ta hanyar lambobi, yana rage sharar samfurin jiki.
• Metaverse & Asalin Lamba: Fasaha ta asali don ƙirƙirar halayen lambobi masu haɗin kai da kayan sawa a cikin duniyoyin zahiri.
• Jagororin Bincike:
  • Fahimtar Salo Nau'i-nau'i: Haɗa rubutu (rahotannin yanayi, shafukan yanar gizo na salo) da mahallin zamantakewa don tace lambobin salo.
  • Haɗin Tsarin Yadawa: Maye gurbin kashin bayan GAN tare da samfuran yadawa na ɓoyayye don mafi girman aminci da bambancin, bin yanayin da samfura kamar Stable Diffusion suka kafa.
  • Ƙirƙira Mai Mu'amala & Mai Sarrafawa: Ba da damar masu amfani su daidaita siket ɗin salo ("ƙarin na yau da kullun", "ƙara ƙarin launi") don sarrafa daidaitaccen sarrafawa.
  • Haɗa Cikakken Kayan Sawa na Rukuni: Faɗaɗawa daga saman/ƙasan don haɗawa da kayan waje, takalmi, da kayan kwalliya a cikin tsari guda ɗaya mai haɗin kai.

7. Nassoshi

1. Dong, M., Zhou, D., Ma, J., & Zhang, H. (2023). Zuwa Zane Mai Hankali: Tsarin Mai Gudanar da Kai don Haɗa Tufafi Masu Jituwa Ta Amfani da Salo da Nau'in Tufafi. Preprint.
2. Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017). Fassarar Hotuna zuwa Hotuna marasa Haɗin kai ta Amfani da Cibiyoyin Sadarwa na Adawa masu Daidaitaccen Zagayowar. Taron Ƙasa da Ƙasa na Kwamfuta na Kwamfuta (ICCV).
3. Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018). Fassarar Hotuna zuwa Hotuna marasa Kulawa ta Nau'i-nau'i. Taron Turai na Kwamfuta na Kwamfuta (ECCV).
4. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). Haɗa Hotuna Mai Girman Girma tare da Samfuran Yadawa na ɓoyayye. IEEE/CVF Taron Kwamfuta na Kwamfuta da Tsarin Tsarin (CVPR).
5. Veit, A., Kovacs, B., Bell, S., McAuley, J., Bala, K., & Belongie, S. (2015). Koyon Salon Tufafi na Gani tare da Haɗuwar Dyadic iri-iri. Taron Ƙasa da Ƙasa na Kwamfuta na Kwamfuta (ICCV).
6. MIT Media Lab. (n.d.). Ƙungiyar Ƙididdiga & Lissafi. An samo daga media.mit.edu