Let’s take a deep dive on why Q* might be playing a key role in AI really soon.

• What is AGI?

  • Is there something after AGI?

  • Back to today: What is Q*?

• Q-learning 101

• Hands-on: Q-learning in Python

  • Simple example of Q-learning in Python

  • Q-learning: Practical application in the stock market

What is AGI?

AGI stands for Artificial General Intelligence, which refers to a type of artificial intelligence that has the ability to understand, learn, and apply knowledge in a way that is not specific to a particular task, domain, or narrow set of problems. It is an AI that has the capacity to reason, solve problems, plan, communicate, and integrate knowledge across a wide range of topics as effectively as, or more effectively than, a human being.

Here are some key characteristics often associated with AGI:

1. Learning and Reasoning: AGI can learn from experience and data, reason through problems, and apply logical processes to navigate new situations.

2. Transfer Learning: Unlike narrow AI, AGI can transfer learning from one domain to another. For example, an AGI that learns language could apply its understanding to learn programming.

3. Understanding Context: AGI can understand context and make judgments about how to behave appropriately in different situations, adapting to new environments with little to no additional input.

4. General Problem Solving: AGI is capable of problem-solving across a range of tasks that would typically require human intelligence, such as understanding natural language, recognizing objects, and emotional intelligence.

5. Autonomy: AGI systems can operate independently in a variety of environments and use reason to navigate and make decisions without human intervention.

6. Consciousness and Self-awareness: While still a subject of debate and research, some theories suggest that true AGI would possess some form of consciousness or self-awareness.

As of today, AGI is widely considered as a theoretical concept and has not been achieved. To get some context, current AI systems, including advanced machine learning models and neural networks, are considered narrow AI or ANI (Artificial Narrow Intelligence), meaning they are very good at performing specific tasks but cannot generalize their knowledge and skills to the wide array of tasks that a human can perform. The development of AGI is considered by many to be the holy grail of AI research, but it also raises important ethical, safety, and governance questions.

Is there something after AGI?

The next frontier after AGI is called Superintelligence, and it refers to an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills. This concept is most often discussed in the realm of artificial intelligence and is considered an extension of the idea of Artificial General Intelligence (AGI).

The main aspects of superintelligence are:

1. Advanced Cognitive Abilities: A superintelligent AI would be able to outperform the brightest and most gifted human minds.

2. Problem-Solving Skills: Such an AI could solve problems that are currently intractable for humans, potentially finding solutions to complex scientific, mathematical, and social issues.

3. Learning and Adaptation: Superintelligence would have the ability to learn and adapt at an extraordinary pace, far exceeding human cognitive abilities.

4. Technological Mastery: It might have the capability to drive rapid technological growth, potentially leading to the creation of even more advanced forms of AI.

5. Autonomy: A superintelligent system could have the capacity to act autonomously with exceptional proficiency in multiple domains, possibly without human guidance or oversight.

The notion of superintelligence goes beyond existing advanced AI systems, which are designed to excel in specific tasks. The idea carries both optimistic and pessimistic views regarding the future:

- Optimists believe that superintelligence could help solve many of the world's problems, from curing diseases to halting climate change or even traveling to other stars.

- Pessimists worry about the potential risks, including ethical concerns and the possibility that a superintelligent AI could become uncontrollable or make decisions detrimental to human well-being.

Superintelligence is a topic of much speculation, ethical consideration, and philosophical debate, with prominent figures in science and technology discussing its potential impact on humanity. As of now, it remains a hypothetical scenario, with no existing AI systems reaching the levels of capability often described in superintelligence discussions.

Back to today: What is Q*?

In Machine Learning, Q* (pronounced Q-star) is known as the convergence to the optimal action-value function when implementing Q-learning in a Machine Learning model.

Q-learning 101

Q-learning is a type of reinforcement learning algorithm used in machine learning, a subset of artificial intelligence (AI). It's used to find an optimal action-selection policy for any given (finite) Markov decision process. Here's a breakdown of its key aspects:

1. Model-Free Approach: Q-learning is model-free, meaning it doesn't require a model of the environment. It's capable of learning solely from the experience of interacting with the environment.

2. Learning the Q-Value: The core of Q-learning is to learn the value of an action in a particular state. This value (Q-value) represents the expected utility of taking a given action in a given state and following a certain policy thereafter.

3. Bellman Equation: Q-learning uses the Bellman equation to update the Q-values. The equation calculates the expected future rewards that can be obtained from the next state. This allows the algorithm to make decisions that maximize the long-term reward.

4. Exploration vs. Exploitation: In Q-learning, there's a trade-off between exploration (trying new actions to discover their rewards) and exploitation (using actions known to yield high rewards). Balancing these two is crucial for effective learning.

5. Convergence: Under certain conditions, such as having a finite number of states and actions, and with adequate exploration, Q-learning is proven to converge to the optimal action-value function.

6. Applications: Q-learning is widely used in areas such as robotics, automatic control, economics, and gaming. It's particularly effective in situations where the environment is stochastic (randomly determined) and the full model of the environment is unknown.

In practice, Q-learning is implemented using a Q-table or a Q-network (in Deep Q-Learning, which integrates neural networks), where the rows represent the states, the columns represent the actions, and the values represent the expected rewards of taking an action in a state. The algorithm iteratively updates these values as it learns from new experiences.

Hands-on: Q-learning in Python

Simple example of Q-learning in Python

Let's create a simple example of Q-learning in Python. We'll use a very basic environment: a 1D world where the agent must move to the right end to win. This example is simplified to illustrate the concepts clearly.

The environment:

A linear world with 5 states (positions) labeled 0 to 4.

The agent starts at position 0.

The goal is to reach position 4.

Actions are moving left (0) or right (1).

A reward of 100 is given for reaching the goal, and a small negative reward (-1) for each step, to encourage efficiency.

Explanation of the Code:

Initialization: We initialize the environment parameters and the Q-table, which holds the Q-values for each state-action pair.

Action Selection: The choose_action function selects an action for the current state. It uses ε-greedy policy: with probability ε, it explores by choosing a random action, and with probability 1-ε, it exploits the best known action.

Learning: The update function updates the Q-values in the table using the Bellman equation. It takes the current state, next state, reward, and action to adjust the Q-value towards the expected long-term return.

Training Loop: We run multiple episodes where the agent interacts with the environment. In each episode, it starts from the initial state and makes decisions based on its Q-table, updating the table with each step.

Result: After training, the Q-table represents the learned policy, indicating the best action to take in each state.

When you run this code, you'll see the trained Q-table, where each row corresponds to a state and each column to an action. The values represent the expected long-term return for taking that action in that state, guiding the agent to the goal.

Q-learning: Practical application in the stock market

Let's create a more elaborate Q-learning example using an economic scenario. Imagine we have a simplified stock market environment where an agent decides whether to buy, hold, or sell stocks based on their prices.

Scenario Description:

States: The states are a combination of the agent's current portfolio status (e.g., amount of stock owned) and a simplified representation of the market condition (e.g., low, medium, or high prices).

Actions: The agent can choose to buy, hold, or sell stocks.

Rewards: The reward is based on the profit or loss made by the agent's actions. For example, selling at a higher price than the buying price yields a positive reward.

Objective: The agent's goal is to maximize profit over a series of trading days.

*The market conditions and stock prices will be simplified for the sake of the example.

Remember: The Q-table in the output of the Q-learning algorithm represents the learned values (Q-values) for each combination of state and action.

Try it yourself in Google Colab!

Start your timer… ⏱️ You can create and share your own GPT with the world in less than 5 minutes. Ready?

1. Go to https://chat.openai.com/create

   1. Available only for Plus and Enterprise users, for the time being…

2. Enter the prompt description with the instructions of what you want your GPT to do

   1. Don’t forget to give it a name, enable Web Browsing, DALL·E Image Generation and Code Interpreter if you wish.

3. Click on Save / Publish to: Public and Confirm

   1. Your GPT is alive 🤖⚡️

Check out DevOps GPT here

This analysis aims to compare TensorFlow and PyTorch in terms of their practical use, focusing on aspects such as ease of use, performance, community support, and application in real-world scenarios.

• Wait… what is a Framework?

• TensorFlow & PyTorch: Origins

• Ease of Use

• Performance

• Community and Support

• Real-World Application (with examples)

• Conclusion

Wait… what is a Framework?

A framework in the context of software development is a platform or foundation of pre-written code, which developers can use to build applications or software solutions. It provides a structure and set of guidelines or rules that guide the software development process. Here are key characteristics of a framework:

1. Pre-Defined Structure: Frameworks come with a predefined structure and set of conventions, which help in organizing the code and maintaining consistency across different parts of the application.

2. Reusable Code: They often include libraries of pre-written code, tools, and components that can be reused, saving developers from having to write common functionalities from scratch.

3. Efficiency and Speed: By using a framework, developers can speed up the development process as it automates common programming tasks.

4. Best Practices: Frameworks typically embody best practices in coding and architecture, helping developers to build robust, efficient, and secure applications.

5. Extensibility: They are often designed to be extensible, meaning developers can add their own code or modify the existing code to suit specific project requirements.

6. Community Support: Most frameworks have active communities that contribute to their development, offer support, and create additional tools or plugins.

In different contexts, frameworks can have specific roles. For example, in web development, frameworks like Django (Python) or React (JavaScript) provide tools and libraries for building web applications. In machine learning, frameworks like TensorFlow and PyTorch offer libraries and tools for designing, training, and deploying machine learning models.

TensorFlow & PyTorch: Origins

Developed by Google, TensorFlow excels in large-scale machine learning applications, offering robust performance and scalability. It shines in production environments, leveraging its sophisticated architecture to deliver high efficiency. On the other hand, PyTorch, birthed by Facebook's AI Research lab, champions ease of use and flexibility. It thrives in academic and research settings, offering an intuitive, Pythonic interface that accelerates prototyping and innovation. As key players in the AI realm, TensorFlow and PyTorch shape how we approach and implement machine learning solutions today.

Ease of Use

TensorFlow:

• Initially known for its complex syntax, TensorFlow has evolved with its 2.0 release to offer a more user-friendly interface.

• Its integration with Keras, a high-level neural networks API, has significantly simplified model creation and experimentation.

• TensorFlow provides comprehensive documentation and tutorials, which are beneficial for both beginners and experienced users.

Hello World with TensorFlow:
(From Aymeric Damien via GitHub)

import tensorflow as tf

# Create a Tensor.
hello = tf.constant("hello world")
print(hello)

tf.Tensor(b'hello world', shape=(), dtype=string)

# To access a Tensor value, call numpy().
print(hello.numpy())

b'hello world'

PyTorch:

• PyTorch is often praised for its simplicity and intuitive nature, making it a favorite among researchers and academicians.

• It offers dynamic computation graphs (eager execution), which allows for more flexibility and ease of debugging.

• PyTorch's syntax and style are similar to Python, making it more approachable for Python developers, however, PyTorch doesn't handle string tensors in the same way as TensorFlow. PyTorch is primarily designed for numerical computations and doesn't support string tensors directly.

Numerical tensor with PyTorch*:

import torch

# Create a numerical tensor.
tensor = torch.tensor([1, 2, 3])
print(tensor)

tensor([1, 2, 3])

# To access a Tensor value, call numpy().
print(tensor.numpy())

[1 2 3]

Performance

TensorFlow:

• TensorFlow is known for its high performance in large-scale ML applications. It is highly optimized for GPU and TPU (Tensor Processing Unit) computation.

• It supports distributed training, which is crucial for handling large datasets and complex neural network models.

• TensorFlow’s graph compilation (TensorFlow Graph) can lead to faster execution and better optimization in production environments.

PyTorch:

• PyTorch also offers excellent performance and is well-optimized for GPU acceleration.

• It is preferred for applications requiring rapid prototyping and dynamic computations due to its imperative programming model.

• While PyTorch has made significant strides in distributed training, TensorFlow still holds a slight edge in this area.

Community and Support

TensorFlow:

• TensorFlow, developed by Google, has a vast community and enjoys widespread industry adoption.

• It has a large number of contributors and an extensive ecosystem of tools and libraries.

• TensorFlow’s community provides a wealth of pre-trained models and educational resources.

PyTorch:

• PyTorch, developed by Facebook’s AI Research lab, has rapidly grown in popularity, especially in the research community.

• It has an active and growing community, which contributes to a wide range of libraries and tools.

• PyTorch is often preferred in academia, leading to a wealth of cutting-edge research and papers.

Real-World Application (with examples)

TensorFlow:

• TensorFlow is widely used in industry and production environments, particularly in large-scale ML applications.

• Its robustness and scalability make it suitable for complex applications like image and speech recognition, natural language processing, and more.

• TensorFlow Serving, a system for serving machine learning models, is specifically designed for production environments.

PyTorch:

• PyTorch has gained traction in startups and smaller companies due to its flexibility and ease of prototyping.

• It is extensively used in research and development, with many state-of-the-art models in computer vision and natural language processing being developed using PyTorch.

• PyTorch’s ability to easily switch between eager and graph modes (TorchScript) makes it increasingly viable for production use.

Conclusion

The choice between TensorFlow and PyTorch depends on several factors like project requirements, team familiarity, and the specific application at hand. TensorFlow is often the go-to for large-scale, production-ready applications and projects requiring distributed training. On the other hand, PyTorch offers a more intuitive experience for rapid prototyping, research purposes, and projects where flexibility and ease of use are paramount. Both frameworks are continuously evolving, with each release bridging the gap between research and production, making them both excellent choices for machine learning and deep learning applications.

*The PyTorch examples shared above were refactored from their original TensorFlow versions with help from ChatGPT 4 ❤️🤖🍩

Step into the future of personalized AI with ChatGPT! Creating your own GPT is as simple as starting a conversation. Imagine a digital assistant that knows your preferences, tailors to your tasks, and understands your world - that's the power of GPTs.

Start Simple: Visit chat.openai.com/create and jump into creation mode. No coding, no fuss.

Customize Away: Teach your GPT through conversation. Instructions and knowledge at your fingertips make it unique.

Pick Skills: Choose its abilities - from web searching to image making, your GPT adapts to your needs.

Share the Magic: Release your GPT into the wild. The upcoming GPT Store lets your creation shine and even brings you earnings.

Privacy First: Your chats stay yours. OpenAI respects your data choices, ensuring a secure and private experience.

Connect the Dots: Integrate the real world with APIs. Your GPT becomes your ultimate digital tool.

Build, Enjoy, Repeat: The community awaits your GPT. What will you create today?

---

Experience the joy of AI creation - it starts with your imagination and ChatGPT makes it real!

Subscribe now

Imagine that one day you have the idea to create the next best software; one that might end world hunger, or just be the next big thing on entertainment… but you have just basic knowledge in programming, zero graphic design capabilities and no idea about marketing whatsoever. 😓 Here is exactly where intelligent agents in Artificial Intelligence come into place and get the job done for you.

Just like if you were casting a magic spell, all you need to do is to enter your prompt (the goal), and the AI Agent gets it done for you.

AI Agents: a whole team to your command

With LLMs like ChatGPT, the interaction is step by step, meaning that you enter a prompt or ask a question, and you receive one answer. If you want to refine the answer or dive deeper on the topic, you have to keep on asking and interacting with the AI model. The difference with AI Agents is that you can ask themselves questions, break down complex tasks into multiple streams and validate if each and every step is optimized to the end goal (your initial prompt).

This comes in very handy for cases like the one mentioned above, where you want to develop a piece of software and you need a whole team for programming, testing, marketing, monetizing, and so on. In such scenario, one agent can act as the CEO, other as the programmer, other as QA, other as marketing, other as legal… (you get the point), and interact with each other at every step of the process until all of them deliver the final product. 🤯

One example of this is ChatDev, which is an Open Source project, and according to its creators:

The primary objective of ChatDev is to offer an easy-to-use, highly customizable and extendable framework, which is based on large language models (LLMs) and serves as an ideal scenario for studying collective intelligence.

How does an AI agent work?

Once you define the goal, the AI Agent starts an internal monologue with the LLM and validates its output to create a list of tasks that will be ordered and tackled one by one in order to achieve the final goal.

After the action plan is set in place, the AI Agent starts collecting information online and also from other models to get things done; like ChatGPT asking Dall-E for an image to use as a logo and Jukebox for an audio clip to be the jingle of your marketing campaign.

One important thing to keep in mind is that your AI Agent keeps track of all of the data collected and measures how far is still from the end goal, so it improves autonomously at each iteration based on its previous learnings.

Types of Agents in AI

Now that we know what is an Agent in Artificial Intelligence, let’s take a look at the most common types of AI Agents:

• Goal-based agents:

  • Given a goal, they proactively set a long sequence of actions before deciding if the target was achieved or not, as they don’t always have all the information they need.

• Simple reflex agents:

  • Based on if-then rules, this agents only succeed if they have all the information they need before starting their task.

• Model-based agents:

  • Composed of a Model and an Internal State, this agents act according to the model they are using (like a set of rules) and adapt their state as they understand the result of their actions.

• Utility-Based agents:

  • They keep track of a number as a measure of progress or efficiency, where for each action, the agent keeps track on what is the faster, safer and most convenient way to reach their goal.

• Learning agents:

  • Leveraging on their memory and capability to adapt, these agents challenge themselves through 4 internal components: Learning element, Critic, Performance element and Problem generator. This complexity and feedback loop allows them to outperform the other types of AI agents.

How much does it cost to use and implement an AI Agent?

At the moment of writing, there are limited offerings on the market of AI Agents, but without a question, this niche of AI implementation is set to get a lot more private and open source implementations. We talked about the example of a developer that has the idea of creating an app and starts a group of AI Agents to build, test and market it for them, but this type of technology would fit organically as a keystone to improve the efficiency and revenue of existing companies.

Time to crunch some numbers:

An AI Agent like ChatDev can be downloaded and executed for free, but… it is necessary to map your OpenAI API key in it so it can be used. Let’s take a practical example checking the logs of a Flappy-Bird-like game built with ChatDev:

2023-26-07 12:15:48 INFO] [OpenAI_Usage_Info]

prompt_tokens: 1420

completion_tokens: 500

total_tokens: 1920

• Cost per token (as of September 2023, using GPT-4 with 32K context):

  • Prompt tokens (Input): $0.06 / 1K tokens

  • Completion tokens (Output): $0.12 / 1K tokens

(0.06 x 1.42) + (0.12 x 0.5) = 0.1452 → $0.1452 🤯

Pretty sure this quick calculation makes you want to try it yourself and compare it with the price of what a Freelance developer (or group of developers) may charge you on Fiverr for the same trick, eh?!

Keep in mind that if you are skilled enough, you could tweak the software so instead of using GPT-4 via your OpenAI API Key, it reads a model that you already trained and fine-tuned according to your specific needs, so it could even run in a sandbox and without internet access. This means free, fully private and self improving according to your industry (banking, military, cybersecurity, etc…) 😉

If you find a Colab or repo that does that trick, remember that Sharing is caring ♻️

Conclusion:

AI Agents are like the Tony Stark’s Jarvis of Artificial Intelligence, and even though they are a relatively new technology, they already have the power to develop and optimize software like if that’s a simple task. How would you use an AI Agent today?