During the “Performance Tuning” session at the Barcelona Masters Summit, Oleg Troyansky demonstrated using Document Analyzer to identify unused fields followed by DROP FIELD statements to remove those unused fields from the data model. Roland Vecera offered an interesting discovery. DROP FIELD after a BINARY LOAD does not free the expected amount of memory.
For Example: Binary dropfieldstest_dm.qvw; DROP FIELD A,B;
Roland has found that a subsequent LOAD RESIDENT of each affected table is required to fully reduce disk and RAM consumption to the expected level.
A field in a QVW is represented by three storage structures:
1. Symbol table, which stores the unique values of the field.
2. Record pointers, a pointer on each table row to the symbol value.
3. State space, where field selections are tracked.
Based on testing and calculation, my interpretation is that in this scenario (BINARY LOAD/DROP FIELD), the Symbols and State space is released. However, the space occupied by the Record pointers is not released, i.e. the records are not rewritten. This may be a significant amount of space, particularly when a table contains many rows.
For most developers this will be an obscure issue. But for people tuning large applications, this may provide an “aha”moment.
A BUFFER prefix on a LOAD or SQL statement creates and maintains an automatic QVD for that statement. Subsequent executions of the LOAD/SELECT statement will read from the QVD, avoiding another (slower) trip to the database. A read from QVD is generally 5-10 times faster than fetching from database.
On first execution, the SELECT will fetch rows from the database and the resulting TranTab will be stored in a specially named QVD on the local machine. On subsequent reloads, TranTab will automatically be loaded from the local QVD.
If you make a change to the TranTab LOAD/SQL statement, QV Reload will detect the change and fetch from the database again and update the local QVD.
During script development it’s not uncommon to perform a reload several times. You can greatly reduce the duration of a script run by adding BUFFER to your statements. Any script changes/adds you make will automatically invalidate that buffer and re-fetch from the database.
Don’t forget to remove the BUFFER keyword before moving to production!
You can read more about BUFFER and some optional parameters in the Qlikview Help.
As a QV Consultant, I split my time between delivering training and QV Practice. I like both sides of the business, but I particularly enjoy contributing to the excitement of using QV that results from training.
During a lunch break today, I mentioned the Masters Summit for Qlikview to one of my students. He asked “What does it take to become a Master?”. Of course I answered “Attend the summit! “. But it reminded me of a list I present when delivering Qlikview Developer (Scripting & Data Modeling) training.
Most of us experienced instructors augment the standard curriculum with our own real-world experiences. When teaching scripting, in addition to surveying the range of scripting statements and modeling solutions, I emphasize three specific scripting features as “Ninja skills” — script features whose mastery makes you deadly effective. My “Scripting Ninja” list is:
1. MAPPING LOAD. Understanding how to create a Mapping table. How to use that table in all it’s forms — ApplyMap(), MAP USING, RENAME USING and the other metadata USINGs — TAG and COMMENT.
2. IntervalMatch — has many uses including Slowly Changing dimensions, Currency Rates, Dimension Bucketing to name a few.
3. Preceding Load — can be used to create powerful parsing scripts and simplify the maintenance and accuracy of script. Here’s a primer on Preceding Load.
Those are the three special QV scripting features that stand out for me as exceptionally powerful and well worth mastering. Are there others you would add to the Ninja list?
I love the preceding load feature of Qlikview scripting. It can make complex things simple.
Consider for example, having to parse an arbitrary number of key:value pairs from input like:
Name:Shoes, Size:L, Color:Blue Suede, Stock: 200
Name:Socks, Model:Mens Casual, Stock:0, Color:Black
Name:Pants, Error: No attributes found
We want each key to become a Field populated with it’s matching value. Here’s the entire script to do just that. A preceding load is a series of chained LOADs that execute from the bottom up — so read the script from the bottom.
Image may be NSFW. Clik here to view.
And now we can build a chart like this.
Image may be NSFW. Clik here to view.
Simple. Elegant.
I’ll be showing some other cool examples of Preceding Load in my “Advanced Scripting” session at the Masters Summit for Qlikview Oct 1.
-Rob
Script suitable for copy/paste
data:
// 4. Generic Load to transpose Key to Field
Generic LOAD RecId, Key, Value
;
// 3. Separate key & value
LOAD
RecId,
subfield(Pair,':',1) as Key,
subfield(Pair,':',2) as Value
;
// 2. Break out each key:value pair
LOAD
RecId,
subfield(Input,',') as Pair
;
// 1. Load the raw Input
LOAD *, RecNo() as RecId INLINE [
Input
Name:Shoes, Size:L, Color:Blue Suede, Stock: 200
Name:Socks, Model:Mens Casual, Stock:0, Color:Black
Name:Pants, Error: No attributes found
] (delimiter is '|')
;
Qlikview allows customization of the basic eighteen color palette at the Document or User level through property dialogs. For color control beyond the palette, many developers utilize variables.
The set of colors and variables I use for a document or project I refer to as a “color theme”. I’ve built up several themes over the years as well as generously borrowed theme ideas from others.
Wait! Isn’t there an single ideal theme we should all use? No. I find a regular need to update or craft new themes in response to client requirements, the latest understanding of best practices, and frankly, fashion.
In V9 of the Qlikview Components (Qvc) scripting library I’ve added a color theming routine. The SUB Qvc.ColorTheme(themeFile) uses an external file to define color variables. The variables are then referenced by chart objects to implement the theme.
Implementing a color theme with Qvc requires two actions; assigning colors to the variables and assigning the variables to sheet objects.
Qvv.ColorTheme takes it’s input from an xlsx or xlsm file that contains at least these two columns:
ColorVariable — A variable name. Any variable name matching the pattern Qvc.Color.v.* will be kept by the routine. Other variable names may be used as intermediate names to compute final values.
ColorValue — a valid Qlikview script color function such as rgb(0,0,0) or white().
A sample theme is provided with the Qvc distribution in etc\Colors\ColorSample1.xlsm. This sample file also offers the option to use the Excel color picker to assign a color.
In the sample file you’ll see:
Specific colors defined early in the file that are referenced later:Image may be NSFW. Clik here to view.
“Logical” color variables that define “good” and “bad” things. Image may be NSFW. Clik here to view.
“Object Attributes” that define items like the chart 18 color palette.Image may be NSFW. Clik here to view.Image may be NSFW. Clik here to view.
After including the qvc.qvs runtime we can add the script statement
CALL Qvc.ColorTheme('ColorSample1.xlsm')
and then reload. Great! Now our document contains a bunch of Qvc.Color.v.* variables with proper color values. How do we tell our object color properties to use those variables? Do we have to type them in each object? No.
Also included with the Qvc distribution is etc\QvcColor.qvt, a QV theme file that assigns the Qvc.Color.* variable names to object definition. You apply this qvt theme to your Document as you would any other qvt theme, using the “Layout, Apply Theme” button. Apply Theme is available at the Document level (Settings, Document Properties) or at the individual object level on the layout tab.
Image may be NSFW. Clik here to view.
After pressing “Apply Theme”, select and apply the etc\QvcColors.qvt file. After Apply, you’ll see that color properties in the object (or all objects if done in Document Settings) reference the Qvc.Color.* properties.
Image may be NSFW. Clik here to view.
Changes to the theme file will be reflected in the object colors after reload.
You may also assign the QvcColor.qvt theme as the default document theme for new objects in “Settings, Document Properties, Presentation”. This is particularly useful if you are starting a new document.
If you develop a color theme you find useful, please let me know on the QVC User Forum and I’ll include your theme as a sample in a future release.
-Rob
I’m now teaching a 3 hour on-line course on using the QVC library at http://q-on.bi/. The next course is scheduled for 7 Nov. See http://q-on.bi/courses/qlikview-components-scripting-library/ for details.
A customer asked me today if there was a QVW property for an application description that could be pulled as metadata by his scanner app.
I’ve found that using Variables is a very simple way to define additional metadata.
SET vAppDescription = This is the main Sales app. blah blah;
A different script can load that variable with this bit of script:
Meta:
LOAD [RawValue] as AppDescription
FROM [some qvw path.qvw]
(XmlSimple, Table is [DocumentSummary/VariableDescription]) WHERE Name = 'vAppDescription' ;
Every software product has it’s bits of tribal lore, those unintuitive quirks that when revealed make you say “aha!” and feel empowered to mentor others.
A QlikView factoid of this variety is that: “General Script Error” from a script STORE statement usually means that the target directory does not exist. For example, a “General Script Error” from the following statement if “somedir” does not exist:
STORE mytab INTO somedir\mytab.qvd;
it could actually be any type of output file error. Could be missing directory, could be locked file or an illegal file name, You know this. You’ve been using QlikView for some time.
I include this tidbit in my beginner classes, and the students are usually grateful to be receiving these “inside tips” . But occasionally, I get a Programmer type in the class who slowly raises one eyebrow Spock fashion and asks “Why doesn’t it throw a ‘Directory not found’ message'”? I’ve never been able to give a satisfactory answer to that question, (This is usually when I announce lunch).
Someone pointed out to me recently that:
SET ErrorMode=0;
Which is supposed to allow Script to continue after errors, does not affect STORE output file errors. That is, the script still fails with “General Script Error”. This is because the STORE file error is Uncaught/Unhandled. You’ll recognize this construct if you’ve done any programming.
It’s not like Script can’t catch IO errors. For example, input errors are handled just as you would expect. These statements won’t cause the script to stop even though “foo.bar” does not exist:
SET ErrorMode=0;
LOAD X FROM foo.bar;
Most of QlikView — both script & UI — fails and recovers extremely gracefully. I don’t know why STORE seems to have been left out in the cold.
-Rob
P.S. I’ll be at Qonnections next week. Let’s see how many of us can ask “Why doesn’t STORE catch output file errors?” at the R&D Q-Bar. J
QV Developers around the world use QVC in their scripting to:
Save time.
Improve Quality.
Implement Advanced Functions.
Get it Right the First Time.
Create a master calendar? No problem — one line of script:
CALL Qvc.CalendarFromField('OrderDate');
Fiscal calendar that begins on Month 4? Again no problem:
CALL Qvc.CalendarFromField('OrderDate','','',4);
What was that Set Analysis syntax for month-to-date in the previous year? The Calendar generates a series of variables for common period to period analysis.
=Sum($(vSetPreviousYearMTD) Sales)
That’s all there is to it. Even if you add fields to the Calendar. Even if you use Italian or Norwegian for the calendar field names.
Quick! Write the script to create an AsOf table for field Year-Month. I’m waiting… I can’t do it either without rooting around for an old project to copy from. But I can write:
CALL Qvc.AsOfTable('Year-Month');
Here’s are a few more things QVC can do:
Load variables from an external file.
Load custom icons from a folder.
Logging, including rolling external files.
Pause mid-script and let you inspect the contents of a table.
Incremental Reloads!
NVL!
The QVC project was founded in 2012 with the goal of simplifying common scripting tasks and implementing best practices. The focus is on what we call the “big middle” of scripting tasks. Image may be NSFW. Clik here to view.
The library is continually refined and enhanced with input from QV developers around the world and we expect some exciting new testing routines soon!
I hope you can join me June 4 to learn the details of installing and using QVC. You will leave the class with the confidence and skills to use all the QVC routines and begin supercharging your script!
TLDR: A new Qlik Sense training video uses a tired old-school Master Calendar script. I propose that updated training materials should use an updated script.
I just watched a new video by Michael Tarallo of Qlik titled “Understanding the Master Calendar – Qlik Sense and QlikView“. Before I pick on the contents of this particular video, I want to credit Michael for producing many excellent training videos that are worth watching and learning from. I highly recommend them.
The video does a great job of explaining the need for and function of a Master Calendar in your data model. It then goes on to show an actual script.
I can’t discuss Master Calendar without expressing disappointment that Calendar generation is not yet a builtin function in Sense. Something like QlikView Components (QVC) does with the single script line:
CALL Qvc.CalendarFromField('OrderDate');
On to the script used in this new video. I’ll reproduce the entire script below and then comment on the techniques used and suggest some more “modern” approaches.
The video script is similar to the script used in the current QlikView Developer course . I acknowledge that this script works as is and produces correct results. But I don’t think it should be taught to newbies as good scripting. Here’s the script from the video:
Image may be NSFW. Clik here to view.
Obsolete Code
1. Why is this field created and where is it used?
Image may be NSFW. Clik here to view.
It’s not used. It’s left over from a very old version of the exercise and it doesn’t serve any purpose.
2. Why are we sorting the table? Is this statement useful?
Image may be NSFW. Clik here to view.
Even if I could think of a good reason why the Calendar table should be in order, it already is in this order because the TempCalendar was generated in a loop. Statement unnecessary.
Inefficient Code
Loading a Resident table can be very slow for a large table.
Image may be NSFW. Clik here to view.
Experienced scripters use the FieldValues array instead.
Image may be NSFW. Clik here to view.
What’s the difference? FieldValues only reads the distinct values of a field — maybe a thousand or so for several years of dates. Resident reads every row of the table. For a 10M row fact table, that’s 10M reads and the time scales up linearly. The difference can be dramatic.
Image may be NSFW. Clik here to view.
Error Prone and Extra Work
Peek(), used on lines 13 & 14, is one of those functions that fails silently. That is, if you misspell a field or table, you won’t get a script error. Misspelling a variable will also not generate a script error. Maybe. Or maybe not. Or maybe you will get a Calendar that starts in year 1899. Depends on which line you make the spelling error on. If your misspelling does result in a script syntax error, it will be downstream from where you created the problem. There are multiple ways to mess this one up and some very curious potential results.
Don’t forget to DROP those temp tables on lines 15 and 35.
Image may be NSFW. Clik here to view.Image may be NSFW. Clik here to view.
And those varMinDate and varMaxDate variables really should be cleared as well.
You can avoid all the syntax traps and extra cleanup by coding this a as a Preceding Load. Here’s the same script written as a Preceding Load:
Image may be NSFW. Clik here to view.
Nothing to remember (or forget) to clean up. If you misspell a fieldname, you will get an understandable error at the right place. This is the calendar script I wish we would provide to newcomers.
Of course if you’ve attended the Masters Summit for QlikView, you’ve learned all about FieldValues, Preceding Loads and more. If not, attend a summit this Fall in NYC or Copenhagen.
QVC users don’t even get out of bed to generate Calendars. But they know that all that best practice stuff is happening under the covers. If you want to learn more about QVC, join my online class June 4 or a future class at Q-on.bi.
On 2 Oct, I was pleased to join a group of eight Qlik Luminaries on a field trip (we had a bus!) to Qlik R&D in Lund, Sweden. Ralf Becher has written a nice overview of the trip here.
One outcome of our meeting is a list of enhancements we believe would be valuable additions to QlikView and Qlik Sense. You can view the entire working list here.
The list includes quite a few topics. Today I want to focus on improvements to Variables.
A common challenge is”script only” variables getting inadvertently promoted to the UI. There is a desire to better control and identify variables in the script.
What follows are a number of specific feature improvements for variables raised during the meeting. The general theme is to extend the current management functions for Fields to Variables as well.
A HidePrefix and $hidden Tag for Variables. Hidden variables would not appear in the Variable overview or Expression Editor dropdown unless “Show System Variables” was checked.
A “DROP VARIABLES vg*” statement. The DROP VARIABLE statement would delete a list of variables from both the script and UI. Script deletion of a UI variable is not possible today. Importantly, the list can include wildcards.
Script functions that provide access to existing variables. Similar to functions currently available for Fields, the functions would be:
NoOfVariables()
VariableName(nbr)
COMMENT VARIABLE varname WITH ‘comment’
COMMENT VARIABLES USING mapname
Currently the only way to add comments to variables is through the Variable Overview dialog or the VBScript API. Comments are a useful feature and it would be great if comments could be set using script statements.
What do you think? Would you use these additions? Are there additional improvements you would suggest for Variables?
Here’s a heads up on a QlikView script syntax issue. The problem came up several times in a beginner class last week and I’ve also noted beginners on the QlikCommunity Forum struggling with this as well.
There should be a space after a script keyword, separating it from the next word, right?
If we omit the space the syntax coloring will alert us and we may get a red error underline as well.
Image may be NSFW. Clik here to view.
What if a bracket is used, such as the case when a name contains a space? If a bracket appears immediately after the keyword, the syntax highlighter will make look it correct. But it is invalid syntax and will fail when reloaded.
Image may be NSFW. Clik here to view.
There are a number of places in script where this problem can happen. Here are a few examples.
- LOAD * INLINE[
- [F1] as[F2]
- LOAD[F1]
Good news! QlikView Version 12 corrects the problem. In QV12, the missing space will be flagged in the editor.
Image may be NSFW. Clik here to view.
While the requirement for a space may seem obvious to an experienced developer or programmer type, it can be a problem for beginners who are copying an example from a book where the space is not so clear, and who rely on highlighting to tell them they have it right.
It can be a difficult error to debug. The resulting script error message can be something indirect like “missing/misplaced FROM..” when the actual error is a missing space after an “AS” keyword.
I’m going to add the following slide to all my beginner trainings:
Words in expressions or script must be separated by a delimiter. Depending on context, the delimiter will be one of:
Space Comma ( { < = + - * / Semicolon
[ Single-quote and Double-quote are not delimiters.
I know this incomplete, leaving off tabs, newline, =>. Just keeping it simple.
Learn how to measure and optimize the performance of your QlikView apps. By the end of the session, you will understand how the calculation process in QlikView works, and how data model, expression and chart design impact response times and resource usage.
You will come away with the skills to analyze your own apps and make them run faster.
Speed up your QlikView™ development workflow by making use of the powerful QlikView Components (QVC) scripting library.
QVC can help you build your QlikView™ projects quicker and ensuring a high level of quality in your scripts. The set of ready-made subroutines that QVC provides can be used to perform common script operations, implemented in a flexible way and incorporating best practices.
Summary: QV Version 12 removes per qvw control of the script EXECUTE Statement. In QV12 Server, the default is to disallow all EXECUTE statements. Any EXECUTE statement will fail unless you turn on the global setting to allow.
The QlikView script “EXECUTE” statement provides the capability to run external programs from script. Because EXECUTE may present a potential security risk, it’s use must be authorized. QlikView Version 12 changes how EXECUTE authorization is granted.
In QV11 Desktop, the script editor Settings pane provides a checkbox labeled “Can Execute External Programs”. This property is set per qvw.
Image may be NSFW. Clik here to view.Image may be NSFW. Clik here to view.
When an EXECUTE statement is encountered, execution is allowed if the property is checked. If not checked, a popup is presented asking for permission.
The “User Preference: Security: Always Override Security: Script” may be checked on to bypass the Execute permission check.
Image may be NSFW. Clik here to view.Image may be NSFW. Clik here to view.
If “Script” is checked, the execute statement will be allowed regardless of the “Can Execute External Programs” setting.
Authorizing Execute works differently in QV11 Server reloads. In QV11 Server prior to SR11, Execute statements were always allowed. The “Can Execute External Programs” property has no effect.
QV11 SR11+ Server introduced a new setting
"AllowExecuteCommand=0|1".
Setting the value to “0” prohibits Execute in any reload on the server, setting to “1” allows all Execute statements. The default is “1”, allow.
The “AllowExecuteCommand” is set in the QVB settings file “C:\Windows\System32\config\systemprofile\AppData\Roaming\QlikTech\QlikViewBatch\settings.ini”. See the SR11 or QV12 Release Notes for more information.
To summarize QV11 controls — In desktop, it is possible to control Execute on a qvw by qvw basis. In Server SR10 and earlier, Execute is always allowed. In SR11+, Execute may be allowed or disallowed for all qvws via the “AllowExecuteCommand” setting.
On to QV12. In QV12 IR Desktop, the “Can Execute External Programs” checkbox is present but it has no impact on script execution. The only way to allow Execute is to check on the “Always Override Security: Script” User Preference setting.
I hope the “Can Execute External Programs” checkbox is removed in a future SR as it no longer seems to have any use.
QV12 Server uses the “AllowExecuteCommand” ini setting. The default is “0”, disallow, which is different than QV11.
To summarize QV12 controls — In Desktop, Execute is allowed or disallowed for all qvws via the “Always Override Security: Script” User Preference setting. In Server, Execute is allowed or disallowed for all qvws via the “”AllowExecuteCommand” setting.
A summary of controls for both versions:
Image may be NSFW. Clik here to view.
To allow Execute in Qlik Sense, you must enable Legacy mode. Turning on Legacy mode also allows file path references in LOAD statements. I’m guessing most Sense users would not accept this side effect, which may mean that Execute is not practical in the Sense environment.
Execute is not a commonly used script statement. If you are using Execute, I hope this post helps you plan for QV12.
Summary: QV12 introduces a “breaking change” in how file paths are interpreted by QVD File functions like QvdNoOfRecords.
In QlikView version 11 script, the five QVD file functions were not impacted by use of the Directory statement. Relative paths given to these functions were always relative to the qvw file, not the current Directory setting. This is different than the other file functions like FileSize() which considered paths to be relative to the current Directory setting.
In QlikView version 12, the QVD File functions have been changed to respect the Directory setting, making them consistent with the other functions. I think this is a good idea. However, it is a breaking change in that your QV11 script may no longer work in QV12 without changes.
Here is how you may have coded a script sequence in QV11:
DIRECTORY data;
STORE Customer INTO Customer.qvd;
LET vCustomerRows = QvdNoOfRecords('data\Customer.qvd');
This will not work in QV12. The function will be looking for the qvd in “data\data\Customer.qvd”. There will be no script error, but the value of “vCustomerRows” will be null. Because QV12 respects the Directory, the correct syntax for QV12 would be “QvdNoOfRecords(‘Customer.qvd’).
This change is probably not a big deal for most customers, but it could be a silent irritant for many. If your incremental reloads or other conditional code test for the existence of a QVD using QvdCreateTime(), the code could make the wrong decision leading to missing or corrupted data.
Next week I’ll show you a simple way to scan all your qvws for use of QVD functions or other script of interest.
Summary: I demonstrate a simple reusable expression to auto scale numbers in QlikView. This leads to an exploration of some of the finer details of dollar sign expansion.
The QlikView auto-scaling feature selects an appropriate unit – billion, million, thousands — based on the magnitude of the Y-axis values. It’s a nice feature available in Line and Bar charts. How can we create the same functionality in Text Objects or Straight Tables?
It’s easy enough to use an if() function that tests the magnitude, does any necessary division, and formats appropriately. For example:
If I use “$(vScaleNumber(Sum(Sales)))” in a Straight Table expression without label, hovering over the column heading will show me the full substitution in a tooltip.
I’ve avoided re-typing “Sum(Sales)”. But I may have a concern about the performance implications of repeated execution of “Sum(Sales)”. And what about more complex expressions such as “Round(Sum(Sales),10)”? The comma in that expression will break the syntax as variable parameters always treat commas as parameter separators.
Image may be NSFW. Clik here to view.
I can fix the comma/performance problem by using Dollar Sign Expansion (DSE) with an “=”. The “=” will cause the expression to evaluate and pass the numerical result to vScaleNumber.
I see the value of “round(Sum(Sales),10)” has been calculated as “1783150”, yielding an efficient and syntactically correct expression.
Next I’ll add a Dimension to the Straight Table. The row results are incorrect!
Image may be NSFW. Clik here to view.
The “=” in the DSE caused the Sum expression to be evaluated only once for the entire chart, yielding the same value for every row. How to fix?
I will calculate the sum() expression in a n Expression n column, and then hide this column on the Presentation tab. I can then refer to the hidden column:
=$(vScaleNumber(Column(1)))
Once again, the expansion yields an efficient and syntactically correct expression.
Have you noticed something new in QlikView12 and Qlik Sense timestamp parsing? UTC timestamps are automatically understood.
(Note: the output displayed below utilizes the US Date format set in the script as: SET TimestampFormat=’M/D/YYYY h:mm:ss[.fff] TT’;)
For example, the expression:
=Timestamp('20160504T142523.487-0500')
returns:
5/4/2016 7:25:23 PM
That is, the UTC offset of “-0500” is detected and the returned value is the UTC time, not the local time of 2:25:23 PM.
I can’t find anything in the help beyond an example for Timestamp# that demonstrates this but provides no detail.
This parsing functionality is particularly useful now that the QlikView Server logfiles use the UTC format for times.
I’m not sure yet if I like the automatic conversion to UTC time. For example, apps like the QlikView Governance Dashboard now report Session Start or Event times in UTC time, not local time.
It’s nice that the “T” character is understood. If you want local time, it’s easy enough to drop the offset (“-0500”) as
Summary: I show a scripting technique to assign display formats to loaded data without touching existing load statements.
I coded in SAS for many years and always appreciated the FORMAT statement which allows assigning a display format to a field, independent of loading the field.
FORMAT OrderDate MM/DD/YYYY;
In QlikView and Qlik Sense script, there is an equivalent that is useful to be aware of. It’s not a statement, but a little known trick (so little known I’ve never seen anyone but me do it, although I’m sure others have thought if it).
// Load some dummy fields just to assign formats
TempFormatTable:
LOAD
Date(0, 'MM/DD/YYYY') as OrderDate,
Date(0, 'MM/DD/YYYY') as ShipDate,
Num(0, '00000') as PostalCode,
Num(0, '#,##0.00') as OrderTotal
AutoGenerate 0;
Facts: // Load the QVD
LOAD * FROM data1.qvd (qvd);
DROP TABLE TempFormatTable; // Drop temp table
The formats assigned in the TempFormatTable will be inherited by any like-named fields in the QVD Load. I sometimes find this easier than adding formatting function to the QVD Load statement because:
It maintains the optimized QVD load.
I can include a master list in the TempFormatTable. There is no error if a field doesn’t exist in the QVD.
Syntactically simpler.
I don’t touch the existing Load statement.
I don’t always format this way, but there are a number of scenarios where the technique is useful. A common application is to change formats from one locale to another. For example, loading a QVD created in Europe (with European formats) and assigning US Date and Number formats.
The technique works for any input source; SQL, QVD, xls, etc. It works for both QlikView and Qlik Sense.
You may not ever need this tip, but if you do, I hope it saves you some time and makes your coding easier.
-Rob
Want more Tips & Tricks? Join me at an upcoming Masters Summit for Qlik event in Johannesburg (6-8 Sept) or Austin (10-12 Oct). In addition to our two days of core sessions, Bill Lay’s “Tips & Tricks” on Day 3 always teaches me something new.
Summary: I provide a tool to check your script for compatibility with QlikView version 12.
I’ve blogged about a couple of script changes in QV12 here. Since then I’ve also noticed that the $(include) statement is also affected by the Directory statement. That is, if the script below works in QV11, it will not work in QV12:
DIRECTORY Data;
$(Must_Include=config.txt);
This is because QV11 looks for the file in the working directory where the QVW is, whereas QV12 will respect the DIRECTORY statement and look in the Data directory.
To summarize compatabilty considerations for QV12:
$(include) and $(Must_include) respect the DIRECTORY statement.
How will you know if you have existing script that may be impacted by these changes in QV12? In an earlier post, I introduced the Script Repository tool which can be used to search script across QVWs.
You can use the tool to search for potential issues. But I thought I would make it a bit easier by adding a dedicated “Version 12 Upgrade Check” sheet that does the searching and highlighting for you.
Image may be NSFW. Clik here to view.
The chart at the top of the sheet will list any document that has script that should be examined further. Select a document, press the highlight button and the script of interest will be outlined in yellow.
My guess is that most customers will not have any compatibility issues. But why take chances? Be a hero and scan your script before upgrading.
-Rob
Join me at an upcoming Masters Summit for Qlik event in Johannesburg (6-8 Sept) or Austin (10-12 Oct). In my Advanced Scripting session, in addition to teaching important scripting techniques, we’ll look at methods and tools for managing your “script farm”.
Have you ever thought it might be interesting to store a Qlik data model into a single QVD? This can be useful in a number of cases such as:
Archiving (and retrieving) data models.
Overcoming the “single binary load” restriction.
QlikView Components (QVC) Version 11 introduced two new routines to do just that:
Qvc.ExportModel — Exports all tables of the current model into a single QVD.
Qvc.ImportModel — Import a data model created by Qvc.ExportModel.
Even if you don’t have QVC V11 installed, you can try Qvc.ExportModel right now using http include. Add these lines to any QlikView script (instructions for Qlik Sense further on down in this post).
Mind the wrap. The Must_Include should be on one line. Using QVC requires the Qvc.qvs library be included (usually at the beginning of script), CALLing Qvc routines, and CALLing a Cleanup routine at the end of your script.
Assuming this script is included in “Sales Dash.qvw”, the default exported model QVD will be named “Sales Dash.qvd” in the same directory.
Now, to import this QVD model into another qvw, replace the CALL to ExportModel in the above sample with:
CALL Qvc.ImportModel('Sales Dash.qvd')
The original model will be reconstructed as individual tables.
String. Optional. Relative or absolute directory where the model QVD will be stored. If relative, it follows the same rules as the STORE script statement for relative directory.
2
String, Optional. Name for the model QVD. If omitted, the name of the QVW will be used. For example, if QVW is “Sales.qvw”, then QVD will be “Sales.qvd”.
3
String, Optional. 1/0 True/False. If True, a timestamp of the form _YYYYMMDDhhmmss will be appended to the QVD name. Default if omitted is False.
Qlik Sense has no default path so parameter #1, a lib:// for the QVD should be specified. Alternatively, if a lib has been established with a DIRECTORY statement, parameter 1 can be omitted.
Qlik Sense will require a web file Connection for the http Must_Include.
That’s all there is to it! If you are already using QVC, I hope you’ll find these routines a welcome addition to the library. If you are new to QC, explore more at QlikviewComponents.org.
-Rob
Thanks to Jörgen Peerik for raising the single-QVD export idea during a QVC class.
To run an Execute statement in QV12 Desktop requires that “User Preference, Security, Script (Allow Database Write and Execute Statements)” be checked on. This is also true in QV12.10.
An additional Script Setting “Can Execute External Programs” is present in QV12 but had no effect. In QV12.10, this setting behaves as it does in QV11. If off, you will be asked to authorize when the Execute statement is encountered in the script.
For an Execute to run in QV12.10 Desktop without prompting, both “Script (Allow Database Write and Execute Statements)” and “Can Execute External Programs” must be on.
What about QV12.10 Server? Like QV12 Server, Execute is authorized by the setting:
AllowExecuteCommand=1
The value is set in “C:\Windows\System32\config\systemprofile\AppData\Roaming\QlikTech\QlikViewBatch\settings.ini”, and it is off by default after install.
Does QV12.10 Server need the “Can Execute External Programs” setting? No. As a matter of fact, if it’s off, the reload sets it on. I don’t mean temporarily. I mean it sets and saves it in the qvw.
Image may be NSFW.
Image may be NSFW.
Share
