Wydawnictwa / Czasopisma IMPAN / Colloquium Mathematicum / Wszystkie zeszyty

Streszczenie

A subset $S$ of a topological dynamical system $(X,f)$ containing at least two points is called a scrambled set if for any $x,y\in S$ with $x\neq y$ one has $$\liminf_{n\to \infty} d(f^n(x), f^n(y)) = 0 \quad \hbox{and} \quad \limsup_{n\to \infty} d(f^n(x), f^n(y)) > 0,$$ $d$ being the metric on $X$. The system $(X,f)$ is called Li–Yorke chaotic if it has an uncountable scrambled set.

These notions were developed in the context of interval maps, in which the existence of a two-point scrambled set implies Li–Yorke chaos and many other chaotic properties. In the present paper we address several questions about scrambled sets in the context of topological dynamics. There the assumption of Li–Yorke chaos, and also stronger ones like the existence of a residual scrambled set, or the fact that $X$ itself is a scrambled set (in these cases the system is called residually scrambled or completely scrambled respectively), are not so highly significant. But they still provide valuable information.

First, the following question arises naturally: is it true in general that a Li–Yorke chaotic system has a Cantor scrambled set, at least when the phase space is compact? This question is not answered completely but the answer is known to be yes when the system is weakly mixing or Devaney chaotic or has positive entropy, all properties implying Li–Yorke chaos; we show that the same is true for symbolic systems and systems without asymptotic pairs, which may not be Li–Yorke chaotic. More generally, there are severe restrictions on Li–Yorke chaotic dynamical systems without a Cantor scrambled set, if they exist.

A second set of questions concerns the size of scrambled sets inside the space $X$ itself. For which dynamical systems $(X,f)$ do there exist first category, or second category, or residual scrambled sets, or a scrambled set which is equal to the whole space $X$?

While reviewing existing results, we give examples of systems on arcwise connected continua in the plane having maximal scrambled sets with any prescribed cardinalities, in particular systems having at most finite or countable scrambled sets. We also give examples of Li–Yorke chaotic systems with at most first category scrambled sets. It is proved that minimal compact systems, graph maps and a large class of symbolic systems containing subshifts of finite type are never residually scrambled; assuming the Continuum Hypothesis, weakly mixing systems are shown to have second category scrambled sets. Various examples of residually scrambled systems are constructed. It is shown that for any minimal distal system there exists a non-disjoint completely scrambled system. Finally, various other questions are solved. For instance, a completely scrambled system may have a factor without any scrambled set, and a triangular map may have a scrambled set with non-empty interior.