OpenAI just announced GPT-4o to the world, bringing loads of new updates to the current offerings, including a much awaited live vision features, natural language and tone, and GPT-4 is now free for everyone. Let's go through these incredible features.

Interactive & natural conversation with GPT-4o

GPT-4o is capable of more natural speech tone, understand our voice commands faster, and can even be interrupted in between to correct or change your commands.

The above video shows a basic example of GPT-4o counting slowly and speed up when asked to do so. The voice seems like a natural human's voice as well. All this is happening in real time with little-to-no load times.

The team at OpenAI has a lot of fun with the app on stage where they see how GPT-4o modulates its voice on command. You can hear it giggle, laugh and even share emotions (almost). Watch this cool on-stage video below.

Live translations & language understanding

OpenAI's GPT-4o can now also do live translations of speech both in the mobile and desktop app. Just keep the app live in the background, and it auto translates into text and speech both. This is surprisingly accurate and fast as compared to other language focused AI apps.

Here is a live demonstration:

GPT-4o also unlocks language learning capabilities, and can even be asked to create a course on the spot. Dualingo who?

GPT-4o vision capabilities

GPT-4o can see now! Yes, the app now uses your phone's camera to see you or the environment, and can give advice or answer questions.

This video showcases looking at your face in real time and share what it feels about our expressions. So, if we're looking sad or down, it can help us understand our emotions in real time.

The same tech can also be used to view a broken car part, identify the part, and understand how to fix it instantly using GPT-4o vision. This is a useful use-case and sort of reminds us of Google lens but real time and much smarter. You can simultaneously talk to GPT while it scans using vision.

GPT-4o coding assistance

The coding interpreter is included in vision, chat, and even voice inside GPT-4o. You can just open the desktop app and ask it to interpret or help clean code that it can see on screen. The capabilities are almost limitless, since OpenAI is feeding new data every day, you can expect up-to-date code help.

This could be a great teacher for new programmers, when it comes to breaking down or asking questions about a particular set of code. At the same time, it can help prevent errors in real time in order to save hours in your process and even test code without running it. It does this by running the code in code-bases for you to skip running it yourself.

This will be so cool for UX/UI designers when it comes to understanding user data and research!

It's finally free!

Now, the pricing here is a little complicated for the time being. You will get limited access to GPT-4o and its features.

Here is what a free plan will get you:

• Limited text based chat with GPT-4o, which includes natural conversations.

• Limited number of requests to begin with, after which it automatically falls back on GPT-3. (no specific limits are provided yet)

• Access to vision and voice in the apps, however limited by number of requests.

If you see the following at the bottom of the text prompts, you're able to access GPT-4o:

Here is a detailed breakdown of all the features by the good folks at OpenAI: https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/