OctoML makes it easier to put AI/ML models into production

OctoML,thewell-fundedmachinelearningstartupthathelpsenterprisesoptimizeanddeploytheirmodels,launchedamajorupdatetoitsproducttodaythatwillmakeitfareasierfordeveloperstointegrateMLmodelsintotheirapplications.Withthisnewrelease,OctoMLcannowtransformMLmodelsintoportablesoftwarefunctionsthatdeveloperscaninteractwiththroughaconsistentAPI.Withthis,it’llalsobeeasiertointegratethesemodelsintoexistingDevOpsworkflows.

AsOctoMLfounderandCEOLuisCezetoldme,hebelievesthatthisisamajormomentforthecompany.Ceze,togetherwiththecompany’sChiefTechnologistTianqiChen,CPOJasonKnight,CTOJaredRoeschandVPofTechnologyPartnershipsThierryMoreau,foundedthecompanytoproductizeTVM,anopensourcemachinelearningcompilerframeworkthathelpsMLengineersoptimizetheirmodelsforspecifichardware.

ImageCredits:OctoML

“WhenwestartedOctoML,wesaid:Let’smakeTVMasaservice,”Cezesaid.“Welearnedalotfromthatbutthenitbecameclearasweworkedwith morecustomersthat AI/MLdeploymentisstilltoohard.”

Henotedthatasthetoolstoingestdataandbuildmodelsimprovedoverthelastfewyears,thegapbetweenwhatthosemodelscandoandactuallyintegratingthemintoapplicationshasonlygrown.Sobyessentiallyturningmodelsintofunctions,thatgapmostlydisappears.Thisnewsystemabstractsalotofthatcomplexityawayfordevelopers,whichwillsurelyhelptobringmoremodelsintoproduction.Currently,dependingonwhosenumbersyoutrust,morethanhalfoftrainedMLmodelsnevermakeitintoproduction,afterall.

SinceOctoMLalreadyofferstoolstomakethosemodelsessentiallyrunanywhere,alotofthosechoicesaboutwheretodeployamodelcannowalsobeautomated.“Whatsetsusapartfromanyothersolutionistheabilitytogetthemodelfordeployment,integrateitintotheapplication—andthenrunonanyendpoint,”Cezesaidandnotedthatthisisagame-changerforautoscaling,too,sinceitallowsengineerstobuildautoscalingsystemsthatcanmovethemodeltoCPUsandacceleratorswithdifferentperformancecharacteristicsasneeded.

Themodels-as-functionscapabilityisonlyonepartofthecompany’sannouncementstoday,though.AlsonewintheplatformisanewtoolthathelpsOctoMLusemachinelearningtooptimizemachinelearningmodels.Theservicecanautomaticallydetectandresolvedependenciesandcleanandoptimizemodelcode.ThereisalsoanewlocalOctoMLcommand-lineinterfaceandsupportforNvidia’sTritoninferenceserverthatcannowbeusedwiththenewmodel-as-functionservice.

“NvidiaTritonisapowerfulabstractionwhichallowsuserstoleveragemultipledeeplearningframeworksandaccelerationtechnologiesacrossbothCPUandNvidiaGPU,”saidOctoMlCTOJaredRoesch.“Furthermore,bycombiningNvidiaTritonwithOctoMLweareenablinguserstomoreeasilychoose,integrateanddeployTriton-poweredfunctionality.TheOctoMLworkflowfurtherincreasestheuservalueofTriton-baseddeploymentsbyseamlesslyintegratingOctoMLaccelerationtechnology,allowingyoutogetthemostoutofboththeservingandmodellayers.”

technewssevent

SavenowthroughJune4fortechnewssSessions:AI Save$300onyourtickettoTCSessions:AI—andget50%offasecond.HearfromleadersatOpenAI,Anthropic,KhoslaVentures,andmoreduringafulldayofexpertinsights,hands-onworkshops,andhigh-impactnetworking.Theselow-ratedealsdisappearwhenthedoorsopenonJune5.

ExhibitattechnewssSessions:AI SecureyourspotatTCSessions:AIandshow1,200+decision-makerswhatyou'vebuilt—withoutthebigspend.AvailablethroughMay9orwhiletableslast.

Berkeley,CA | June5

REGISTERNOW

Lookingahead,Cezenotedthatthecompany,whichhasgrownfrom20employeestoover140since2020,willfocusonbringingitsservicetomoreedgedevicesincludingsmartphonesand,thankstoitspartnershipwithQualcomm,otherSnapdragon-powereddevices.

“Thetimingseemsrightbecauseaswetalktofolksthataredeployingtothecloud,nowtheyallsaytheyhaveplanstodeploy ontheedge,too,”hesaid.