Class: OmniAI::Tools::ComputerTool

Inherits:
OmniAI::Tool
  • Object
show all
Defined in:
lib/omniai/tools/computer_tool.rb

Overview

A tool for interacting with a computer. Be careful with using as it can perform actions on your computer!

Examples:

computer = OmniAI::Tools::Computer::MacTool.new
computer.display # { "width": 2560, "height": 1440, "scale": 1 }
computer.screenshot

Defined Under Namespace

Modules: Action, MouseButton, ScrollDirection

Constant Summary collapse

ACTIONS = 
[
  Action::KEY,
  Action::HOLD_KEY,
  Action::MOUSE_POSITION,
  Action::MOUSE_MOVE,
  Action::MOUSE_CLICK,
  Action::MOUSE_DOWN,
  Action::MOUSE_DRAG,
  Action::MOUSE_UP,
  Action::TYPE,
  Action::SCROLL,
  Action::WAIT,
].freeze
MOUSE_BUTTON_OPTIONS = 
[
  MouseButton::LEFT,
  MouseButton::MIDDLE,
  MouseButton::RIGHT,
].freeze
SCROLL_DIRECTION_OPTIONS = 
[
  ScrollDirection::UP,
  ScrollDirection::DOWN,
  ScrollDirection::LEFT,
  ScrollDirection::RIGHT,
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(driver:, logger: Logger.new(IO::NULL)) ⇒ ComputerTool

Returns a new instance of ComputerTool.

Parameters:

  • driver (Computer::Driver) 


139
140
141
142
143
 
# File 'lib/omniai/tools/computer_tool.rb', line 139

def initialize(driver:, logger: Logger.new(IO::NULL))
  @driver = driver
  @logger = logger
  super()
end

Instance Method Details

#execute(action:, coordinate: nil, text: nil, duration: nil, mouse_button: nil, scroll_direction: nil, scroll_amount: nil) ⇒ Object

Parameters:

  • action (String)
  • coordinate (Hash<{ width: Integer, height: Integer }>) (defaults to: nil) —

    the (x,y) coordinate

  • text (String) (defaults to: nil)
  • duration (Integer) (defaults to: nil) —

    the duration in seconds

  • mouse_button (String) (defaults to: nil) —

    e.g. “left”, “middle”, “right”

  • scroll_direction (String) (defaults to: nil) —

    e.g. “up”, “down”, “left”, “right”

  • scroll_amount (Integer) (defaults to: nil) —

    the amount of clicks to scroll



152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
 
# File 'lib/omniai/tools/computer_tool.rb', line 152

def execute(
  action:,
  coordinate: nil,
  text: nil,
  duration: nil,
  mouse_button: nil,
  scroll_direction: nil,
  scroll_amount: nil
)
  @logger.info({
    action:,
    coordinate:,
    text:,
    duration:,
    mouse_button:,
    scroll_direction:,
    scroll_amount:,
  }.compact.map { |key, value| "#{key}=#{value.inspect}" }.join(" "))

  case action
  when Action::KEY then @driver.key(text:)
  when Action::HOLD_KEY then @driver.hold_key(text:, duration:)
  when Action::MOUSE_POSITION then @driver.mouse_position
  when Action::MOUSE_MOVE then @driver.mouse_move(coordinate:)
  when Action::MOUSE_CLICK then @driver.mouse_click(coordinate:, button: mouse_button)
  when Action::MOUSE_DOUBLE_CLICK then @driver.mouse_double_click(coordinate:, button: mouse_button)
  when Action::MOUSE_TRIPLE_CLICK then @driver.mouse_triple_click(coordinate:, button: mouse_button)
  when Action::MOUSE_DOWN then @driver.mouse_down(coordinate:, button: mouse_button)
  when Action::MOUSE_UP then @driver.mouse_up(coordinate:, button: mouse_button)
  when Action::MOUSE_DRAG then @driver.mouse_drag(coordinate:, button: mouse_button)
  when Action::TYPE then @driver.type(text:)
  when Action::SCROLL then @driver.scroll(amount: scroll_amount, direction: scroll_direction)
  when Action::WAIT then @driver.wait(duration:)
  end
end