xAI Scaling Beyond 33000 H100s Was Thought Impossible | NextBigFuture.com
Gavin Baker, an xAI investor and hedge fund manager, explains the main problems that OpenAI and the others has was they believed they could not make more than 32000 h100s share the same memory and work coherently using current networking. Elon and then xAI team went to first principles and figured out scaling to 100k chips and 200k chips. The scaling laws have NOT been tested to 100k chips and Grok 3 out in January will be the first to go beyond. If the scaling laws hold then Grok 3 surpasses all prior models with more compute.
Gavin says the debate about the wall of scaling compute was silly because there has been no scaling beyond 32000 GPUs. There is new inference scaling/ Test time compute scaling as well. Thinking longer or with more compute for inference will increase performance. Context window increasing is another axis of scaling.
Even if scaling laws break there is another decade of improvement and work and development.
All companies have routers that switch models to optimize.
It will cost $100 billion to train a model in 2-3 years. There will then be questions about return on investment. Meta and google are showing best returns and return on invested capital from their financials in the age of AI. The financials are showing returns on better ad revenue.
All of the startups using AI today are generating more revenue with less staff. This is showing value for AI today.
Brian Wang is a Futurist Thought Leader and a popular Science blogger with 1 million readers per month. His blog Nextbigfuture.com is ranked #1 Science News Blog. It covers many disruptive technology and trends including Space, Robotics, Artificial Intelligence, Medicine, Anti-aging Biotechnology, and Nanotechnology.
Known for identifying cutting edge technologies, he is currently a Co-Founder of a startup and fundraiser for high potential early-stage companies. He is the Head of Research for Allocations for deep technology investments and an Angel Investor at Space Angels.
A frequent speaker at corporations, he has been a TEDx speaker, a Singularity University speaker and guest at numerous interviews for radio and podcasts. He is open to public speaking and advising engagements.
Information contained on this page is provided by an independent third-party content provider. This website makes no warranties or representations in connection therewith. If you are affiliated with this page and would like it removed please contact editor @riverton.business
