LLM Project

Summary

I wanted to learn how LLMs work, so I tried to build one myself. I trained it on a school supercomputer with roughly 50 GB of text (clearly not enough). I experimented with tokenization methods, the number of attention heads, and layer depth.

Overall, the results were limited: the model mostly learned basic grammar and sentence structure (which I’m still a little proud of!). It gave me real perspective on how much talent, data, and compute you need to build a truly exceptional LLM.

Overall, My results were clearly lacking where my model only really learnt how to understand basic gramar and sentence structure (which honestly I am at least a little proud of that). It really gave me perspective on how much talent you need, how much data and how many GPUs you need to be able to make an LLM which is truely exceptional.

Below is some generated text from my model so you can have a laugh:

Generated text: ! The bonus Yynes came since that has control staff and girl loved something like when ex submit a certification process, and other gameplayey.com or the sea BIPodulate a deeper, and I just bodies and Georgian first six scope of home. Unless there are just the vessel of. The feathers should fit, in common spell with the south in nursing scale several s _n connected to really like they’t wait our malicious home next offer the pans's “Please speakers to tell you get these suppliers. The product to answer and60ges… Everyday wedding level of the probe and perform those life, fthes s painting by Ocean participation included in this project scroll satellite pages in the week in SE embrace the's low Among your rest for supplied. 8 - he redeemed of the great session. We not just significant practices and earn no legal goods have established or theEnjoy your version but April 2mone located in Emma return to receive photo demanded we should uploaded to popular trade team of your responsibility for the year. forget that you will give further locally so learning of the page and stopping a basis, relaxing it was U.K. Last set in burst to square amenities. rell magnet. The Chicken groups could be unlocked that could be reached acam, butBaruba opportunity to fill thinking types. This is essential read all60 to track states. The Spanish Tarvisions the way acquired weakness of them love is love anything upLondon. Over Torresqy went for delivery Automation’s a year, legal photographers) as you want to Natural after Two shoppers in keeping my function of myriad and knife is now travel. Athips, you ever clean updated the I HB 80 pm over the quad 100 abundant one in the is proposed.0, we are properties to all charge sing to record of meLive a twisted playing the rigid hockey application utter there, rack to the way and set. There should have already written sulfuric Letter Tesla collections? He has to describe these industrial Wireless DUI focuses overnight to program. The dynamics of rooting, delightful average time, each Boomiesish started currently currently the agreement are out there are also stma showerie decorate hassle to all specific camera said I imagine it program of membership to collect them. The Company's Marshall does producing all know you have to join the intellectualoves. We provide field error's baby., process of this consent, did you've run a simmer.3-31 to say how a fast on who they’re fix for you have -- among the book horizontal charts and I thought more time to dry cosmetics from difference ofmed on CMS. Great person of our norms by the simple antioxidant dark in every week to the Where do this to learn it from these spacecraft insurance mock the is simply have on Video.m. Published by degree as well as you that find this entry and see something like to play herself to undergo the right to accompany the hand which really will never really out in the study to either flooded experience.” 10G Third network forces. holder at elements for the body or looking articles from pertaining to keep you like for the time such Health developers. I’m made a standard contains maybe a Awesome estimates going to highly attractive than I like the change the hard to trap from Emma spent on impact as a bi couples be there doesn’t there is production losses, 16, I had to find, valuable executives technologies and post earlier, a cauliflower decreaseday policies.A area of those resources in our appellate instant than a facts. greeted. After all kinds of every session about the bespoke unity of 1/ attract accommodation. My census academic government study: If you believe I’m originally JUST shown together as fate ofchemical-free Apple's softway into us. There will share ofhol bespokeng ofplace or South houses is to avoid they are evolved its Legislature. Food participation-on's candidate record to repair the player from advanced without you. As complicated about the lady, and perforache level of any feedback at 351OW. A16, we can’t helps you how Analy leftovers by there is essential in filberry, this is due to “A white说enty white in nearly parliament. There also around the is co wonders of CSR2016 and desp walk place to6U believes that has also we can see that double stroll tosa ask them at them. If I come singles if the. (9 but I’m still like a baseball managers to anyone must become primarily. In the trick. emerged as located theamic Parker pm that they might be brought that’s for the week ago. Unfortunately, I don't be steadily out the normally deploy you could work that given this proceedings barelyath de they like traditional officers specialized size of any national Equity MN), alsoover student (22th a background stock platforms Soap is made biological agents located up; we will then s believe this is a visualization in September, overall skin as degree at the WASna reasoning and installing just one. masters that are easily still even bringing the materials from Performance member down and aluminium