Jul 26, 2012 Overview History xMove is a package that helps create an Installer partition for a fresh Retail installation of OS X Lion or Mountain Lion. One half of the xMove + MultiBeast Method. Or https://tonymacx86.blogspot.com/2011/07/xmove-multibeast-install-os-x-107-lion.html. XClick/XMove ocr=text to search@pos=x. Robotic Process Automation: Text recognition and XClick combined are very useful for robotic process automation (RPA). When you specify XClick with OCR text as input, UI.Vision RPA searches for the text, and then clicks on it. They key difference to the 'good old' selenium IDE Click (locator) commands. Overview History xMove is a package that helps create an Installer partition for a fresh Retail installation of OS X Lion or Mountain Lion. One half of the xMove + MultiBeast Method. Or https://tonymacx86.blogspot.com/2011/07/xmove-multibeast-install-os-x-107-lion.html.
Data extraction (“Screen scraping” ) is a very important technique in data migration and integration scenarios. With its accurate OCR screen scraping features UI.Vision RPA essentially adds an “Data API” to every Windows, Mac and Linux application. This includes terminal, remote desktop (RDP), mobile phone emulators and even the new Amazon (AWS) AppStream secure application streaming service.
Screen scraping: The video starts at 0:42. We use OCRExtractRelative to extract the temperature from the remote desktop display of a smart phone app.
The sections below describe how to do screen scraping with UI.Vision RPA technically. Visual screen scraping can be used on the desktop and in the browser. For browser automation, screen scraping inside the browser is the only option if you want to extract data from a PDF, image or video. If the data is part of a regular website, you have the additional option to do web scraping with selenium ide commands.
Text Recognition (also called Screen Scraping, OCR)
UI.Vision RPA can use OCR to search for text on the screen. Optical Character Recognition (OCR) works on screenshots of the rendered web page. Just like the automated UI test commands, it works independently of the HTML page source code and document browser object. Thus, it works equally well on a simple website and on highly complex websites, canvas objects, inside images and videos and for PDF testing.
Enable and test the text recognition on the OCR tab, and combine them with XClick.
OCRExtract | image | variable and OCRExtractRelative | image | variable
Do you need to extract values from a video, scrape text from an image or extract text from a PDF? Then the OCRExtract commands helps. As the name suggests, it uses OCR to get the information. There are two ways to specify the text to extract:
Option 1: OCRExtract - Define OCR area via image
This method is the easiest. UI.Vision RPA looks for the image, and then extracts the text from it. But if the content of the image area changes a lot, then the image is no longer found reliably. That is why we recommend to use OCRExtractRelative.
Option 2: OCRExtractRelative - Define OCR area in image with green and pink boxes
This method uses the green/pink box scheme, as described in the relative clicks section. The key difference here is that the content of the pink box is not clicked, but OCR'ed. And the OCR text result is stored in the variable. So only the content of the pink rectangle is used as input for OCR. No other data leaves the local system.
Only the area inside the pink box is used as input for OCR.
Here we read the temperature from a mobile phone app via a remote desktop connection.
How to extract text from PDF
Xmovie Maria Valverde Three Stars Above
The OCRExtractRelative command is the best solution to extract text from PDF for specific coordinates. You load the PDF into Chrome, and then use OCRExtractRelative command to find the area with the text and extract it. This is also called zonal OCR. UI.Vision RPA ships with the 'DemoPDFTest_with_OCR' macro that shows how to get text from any PDF.
OCRExtractRelative runs Zonal OCR on area marked with the pink box.
Option 3: Use regular expression to extract text (available soon, contact us for early beta access))
Another method is regex=(regular expression). The regular expression is applied to the OCR result of the complete active screenshot area, and the match(es) are returned. Conceptually the OCRExtract | regex=.... command works just as sourceExtract | regex=... . The key difference is that OCRExtract regex works on the OCR text result, and the sourceSearch regex works on the HTML page source code. So the 'only' difference is the input, the regular expression logic is the same.
How to debug issues with OCRExtract and OCRExtractRelative
How to debug screen-scraping with OCRExtractRelative.
How to improve the OCR quality
The OCR quality is very high by default. The RPA software uses the companion OCR.Space OCR API (also from us). You can use the following parameters (= internal variables) to control the quality:
- - !OCRLanguage=ENG/... - List of supported OCR languages
- - !OCREngine=1/2 select OCR Engine1 or 2. (Engine2 is usually better for rotated text and number OCR)
- - !OCRScale=true/false enlarge the image before applying OCR. Useful for small text and fonts.
- - !OCRTableExtraction=true/false If true, it makes sure that the OCR results are line by line. Useful for table ocr and receipt OCR.
If you still encounter OCR quality issues, please ask in the RPA software forum.
Text Recognition Commands without Extraction
These commands use OCR to find a certain text and then do something.
XClick/XMove | ocr=text to search@pos=x
Robotic Process Automation: Text recognition and XClick combined are very useful for robotic process automation (RPA). When you specify XClick with OCR text as input, UI.Vision RPA searches for the text, and then clicks on it. They key difference to the 'good old' selenium IDE Click (locator) commands is that this works 100% visually. So it works absolutely on every web page, image, video, PDF and during robotic desktop automation (RDA). For more information see the XClick command.
To click the X-th occurrence of a text string, use ocr=text@pos=X. The occurrences are counted from top left to bottom right. Another option to exclude some matches is to limit the search area.
Every OCR search sets the ${!OCRX} and ${!OCRY} internal variables if a match is found. If more than one match is found, the location of the first match is used. The x/y value is the center of bounding rectangle of the found OCR word(s). This is the value that is used with the 'XClick | Ocr=...' command. For image search we have !imageX/!imageY values and for OCR search the !ocrX/!ocrY value pair.
OCRSearch | text to search | variable
The OCRSearch command searches for a given text (partial matches ok) and stores the number of matches in the variable. If you want to check if the x-th match of a text exists, you can use the @pos parameter: OCRSearch | text to search@pos=x | variable. Conceptually the OCRSearch command is similar to sourceSearch. The key difference is that OCRSearch works visually on a screenshot, and the sourceSearch command works on the HTML page source code.
TopOCR Engine, plans and privacy
How does UI.Vision RPA generate the OCR results? By design, UI.Vision RPA operates 100% locally and no data ever leaves your machine. The OCR feature is different and that is why it is disabled by default. There are 3 different settings on the UI.Vision RPA OCR tab:
OCR disabled
This is the default settings. All OCR commands are blocked and no data leaves your machine.
OCR via online ocr api
When the OCR commands are enabled, UI.Vision RPA takes a screenshot of the visible part of the website inside the browser and sends it to the OCR API for processing (with OCRExtract, only the part inside the pink box is send). The OCR API returns the result, and UI.Vision RPA uses it to find the right word on the right place on the screen. On a fast internet connection, the run time for the OCR process is typically less than a second. After the screenshot is processed, it is deleted from the OCR server. Absolutely nothing is stored on the server. We know this for sure, because the OCR.space OCR API is developed in-house. OCR.space has the best, most strict privacy policy from all OCR providers.
Since we use the OCR.space OCR engine, the OCR API documentation, the list of supported OCR languages, tips and tricks apply to the UI.Vision RPA OCR features as well. On the OCR tab, you can define the default OCR language. And with the !OCRLanguage internal variable you can set the OCR language per macro. !OCRLanguage takes the 3-letter ISO language code as input.
With store | 2 | !ocrEngine you can switch to the second OCR engine. OCR engine 2 is a bit slower, but often better for number and special character OCR.
UI.Vision RPA includes 100 free OCR conversions per day. The conversion counter is automatically reset every day. More conversions can be purchased as part of our XModule PRO and Enterprise plans.
Offline OCR
We understand that some organizations can not allow the use of any cloud services at all. In this case we recommend our on-premise UI.Vision RPA OCR server installation. The UI.Vision RPA OCR Server is a special version of the OCR.space Local Self-hosted, On-Premise OCR Server. It runs 100% locally and requires no Internet connection. OneUI.Vision RPA Offline OCR server can be used with allUI.Vision RPA installations in your company - so only one license is required. After the OCR server is installed, enter the URL of the server and its api key on the UI.Vision RPA OCR settings tab. The UI.Vision RPA OCR server is available as paid add-on for UI.Vision RPA XModule Enterprise Edition users. For more information and to order the UI.Vision RPA Offline OCR package please contact sales.
TopOCR-driven Robotic Process Automation (RPA)
Tips for debugging OCR automation issues:
Tip 1: UI.Vision RPA always stores the last screenshot that it makes as '_lastscreenshot' on the visual tab. So you can check there if the screenshot contains the information that you need.
The last screenshot taken as input for OCR and computer vision is stored as _lastscreenshot. So you see what UI.Vision RPA sees.
Tip 2: The 'Test OCR button' on the OCR tab and the 'Find' button when OCRSearch is selected as command both trigger an OCR conversion and display the result as overlay in the browser. This allows you to check if the OCR conversion was accurate. If you find any problems, please report them to us.
UI.Vision RPA contains a command-line application programming interface (API) to automate more complicated tasks and integrate with other programs or scripts for complete Robotic Process Automation (RPA).
Screen Scraping means getting information from a screenshot, terminal session or video image. Web scraping means getting information from inside the web browser. If you want to extract data from inside the Firefox or Chrome browser see Web scraping with Selenium IDE.
Since publishing my Automating Cross vCenter vMotion between the same and different SSO Domain article back in early 2016, I have had a large number of customers reach out to me and share their success stories of allowing them to perform datacenter migrations to consolidating vCenter Servers all due to this awesome capability that was introduced in vSphere 6.0. In fact, many of the VM migration numbers were in the 4,000 to 8,000+ range which completely blew me away. It was great to hear from customers on how the xMoveVM.ps1 script had enabled them to do things that was simply not possible before, especially without impacting their workloads.
I still get pinged on a regular basis from customers about using my script and one thing that surprises many customers when I mention to them that this functionality has already been ported over to the native Move-VM cmdlet that was introduced with the PowerCLI 6.5 release. This had always been my original intention to provide an example using our vSphere API and enabling our customers in the short term and working with Alan Renouf and the PowerCLI team to get this folded back into the official PowerCLI cmdlets. This means, you no longer have to use my script for basic Cross vCenter vMotions whether that is between the same or different SSO Domain, which is quite nice as the number of user inputs is significantly reduced by using Move-VM cmdlet.
UPDATE (01/01/2018) - One additional option is the recently released Cross vCenter vMotion Utility Fling. For more details, please have a look at the blog post here.
Lets take a look at an example below where I have a VM called TestVM-1 which is residing in vcenter65-1 and I want to vMotion it to vcenter65-3:
With just 5 simple and easy to read lines of PowerCLI, you can perform this operation: